Friday May 16, 2008
Jyri Virkki
Friday May 16, 2008
Web Stack Road Trip
It's been very busy times lately so haven't had a chance to blog..
Thanks to everyone who came to my OpenSolaris AMP (slightly misnamed as it covered all of Web Stack) talk at CommunityOne last week.
Next week I will be at the Sun Tech Days conference in Mexico City again talking about OpenSolaris Web Stack. I hope to see some of you there!
Posted at 08:48PM May 16, 2008 by jyri in WebStack | Comments[0]
Thursday April 24, 2008
Web Stack Package Repository
A little while I ago I introduced the Web Stack Experimental Repository. It's a great way of installing both early access (in development) packages as well as packages that won't necessarily go to any formal integration. The only drawback has been that accessing the repository has required a small patch to pkg (the IPS client).
Great news!
pkg bug 767 has been fixed this week which means it is no longer necessary to modify pkg to install packages from the Web Stack repository!
To recap the now simplified instructions, here's how you can install one of our experimental packages:
1. Make sure you have the latest IPS
1.1 If you are on a Nevada build:
Nevada builds don't have IPS by default so you need to compile and install it:
% hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate % cd gate/src % make % make install % su # make link
The 'make link' target will make symlinks under /usr/bin and elsewhere such that you can just invoke 'pkg' conveniently.
1.2 If you are on Indiana:
As long as you install Indiana RC2 or later you're all set out of the box!
(If you are on an older build of Indiana you could follow the instructions shown above and just get and build the latest from the source tree. The only additional complexity is that since pkg is already part of the system, it might be easier to run it directly from the workspace instead of attempting to 'make link', but that might cause other issues I have not tested. Better to upgrade.)
2. Add the Web Stack repository to your list of repositories:
Assuming you don't already have some other authority called "webstack" (if you do, pick a different name):
# pkg set-authority -O http://pkg.opensolaris.org/webstack webstack
(or prefix the invocation with pfexec if you're not su'd to root)
3. Install something!
Let's say you want to try the experimental Varnish package:
# pkg refresh # pkg search -r varnish INDEX PACKAGE basename pkg:/SUNWvarnish@0,5.11-1:20080408T160306Z # pkg install SUNWvarnish DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 28/28 0.83/0.83 PHASE ACTIONS Install Phase 50/50
4. Enjoy!
Feedback is welcome as always, please join the Web Stack list for any comments. If you'd like to contribute any package to this experimental repository, check with me on the list and I'll point you in the right direction...
Posted at 04:28PM Apr 24, 2008 by jyri in WebStack | Comments[3]
Tuesday April 08, 2008
SFW Over Time
Some weeks ago I was looking around in the SFW sources and, while doing other things, collected a list of the creation dates of each component directory. I ran across that data file tonight and decided to plot it over time to see what it looked like:
Interesting how nearly all the components were created in either 2001, 2004 or 2007 and between each of these growth years there was a two year idle gap.
Since 2007 was a busy year I wonder if we're now going to have two more years of no progress? ;-) Given how many new things are in the pipeline, I very much doubt that!
On the other hand, if we truly succeed in unconsolidating there won't be much reason to keep adding many components into SFW so perhaps the ideal goal is indeed a flat graph for SFW component totals from here on! As I mentioned recently, our experimental web stack repository is the first baby step towards this goal so there is hope.
Posted at 10:49PM Apr 08, 2008 by jyri in WebStack | Comments[0]
Monday March 31, 2008
10 Years
Last time I was doing laundry I realized it is 10 years since the Mozilla source release and party back in 1998. What is the connection to laundry? Well, as with all Silicon Valley history, it is best documented by the trail of event t-shirts ;-)
To help celebrate the occasion, here are pics of the two shirts I have from that night.
I meant to also add a pic of the autographed source CD from the event but didn't find it right now. I'll update this entry with a pic later.
I see http://mcom.com/ is back up as well...
Posted at 07:57PM Mar 31, 2008 by jyri in WebServer | Comments[0]
Friday March 21, 2008
Experimental Web Stack Repository
In several of my previous entries I have commented on the need to have a way to distribute experimental and/or nightly builds of upcoming Web Stack packages so it is possible for everyone to try them out and test the proposed functionality before it is fully baked into OpenSolaris (and thus very hard to change).
I'm happy to announce that today we have such an experimental repository at http://pkg.opensolaris.org/webstack/ (huge thanks to Stephen Hahn for the hosting, bandwidth and setup!)
So, what's in it?
Currently my intent is to make available nightly builds of components under development (though I probably won't update it every night yet). Being nightly builds, at any given time some (or all) of the components might not work at all, so they are only suitable for experimenting.
Some of the components may eventually make it into more formal distribution channels (such as the official part of the Indiana repository) once they mature. Others might remain available only as experiments. Time will tell...
You can certainly influence the decisions by participating on the Web Stack project list. I'm hoping you will also participate by installing these experimental packages and providing feedback on how useful they are and what's wrong with them.
EDITED April 24: It is no longer necessary to patch pkg as described below. Refer to my more recent article on the Web Stack repository for the current info.
How to use it (Nevada/SXDE version)?
1. Obtain the IPS pkg client:
% hg clone ssh://anon@hg.opensolaris.org/hg/pkg/gate
2. Patch IPS. The bad news is pkg doesn't yet know how to properly interact with a repository which is not at / (such as /webstack) so for now you'll need to apply a patch. A patch is available in bug 831 (the final change might be different, but this will get you going). Click on the link "fixes uri construction so path is preserved" in the Attachments section to get the patch and save it to a file pkg.patch.
% cd gate/src/modules/ % gpatch misc.py < ~/pkg.patch
3. Build and install IPS
% cd gate/src % make % make install % su # make link
Notice this will create links from /usr/bin/pkg and other locations into your workspace (so don't delete your workspace). Obviously this is not a very permanent installation... but it makes easy to periodically update the IPS snapshot.
4. Add the Web Stack repository. You can initialize it and make the Web Stack repository your primary source of packages by running:
# pkg image-create -F -a webstack=http://pkg.opensolaris.org/webstack/ /
Note: If you're already using IPS you'll want to add Web Stack as an additional repository instead. Add the following lines to the end of your /var/pkg/cfg_cache file:
[authority_webstack] origin = http://pkg.opensolaris.org/webstack/ prefix = webstack mirrors = None
4. Check out the available packages...
% pkg refresh % pkg status -a FMRI STATE UFIX pkg://webstack/SUNWphp5-myadmin@0-1 known ---- pkg://webstack/SUNWwebstack-build@0-1 known ---- pkg://webstack/SUNWwebstackui@0-1 known ----
(the output will vary depending on your configuration and what's available today)
5. Install something!
# pkg install SUNWphp5-myadmin ...
will get you the experimental phpMyAdmin package.
Limitations
As I mentioned earlier, keep in mind these are fragile experimental packages. On any given day they might not work at all. If you try them out (please do!), send feedback and suggestions to webstack-discuss@opensolaris.org.
Another limitation is that these packages currently have no dependencies set. So it will happily install, for example, SUNWphp5-myadmin even if you don't have Apache or PHP or MySQL installed even though that's not going to work. For now you will need to manually install the dependencies.
Enjoy!
Posted at 03:40PM Mar 21, 2008 by jyri in WebStack | Comments[1]
Wednesday February 06, 2008
Web Stack and NetBeans
While you're trying out the recently released SXDE 1/08 Web Stack components, here's a page from NetBeans with several useful HowTo's: http://wiki.netbeans.org/NetBeansAndSolarisHowTos.
Posted at 06:37AM Feb 06, 2008 by jyri in WebStack |
Tuesday February 05, 2008
Web Stack and SXDE 1/08
SXDE 1/08 has been released today, go check it out!
If you've been reading my blog for the past six months you've seen the progress on including and functionally integrating several web tier components into OpenSolaris. While these have been available for a number of weekly builds now, SXDE 1/08 is the first named release containing all the Web Stack components so it's a nice milestone.
Last week I wrote a HOW-TO (half jokingly, since it consist of only one step ;-) on using PHP with OpenSolaris so that might be a good place to start when experimenting with SXDE 1/08.
For all comments and feedback, join us at the Web Stack project.
Posted at 10:29AM Feb 05, 2008 by jyri in WebStack |
Wednesday January 30, 2008
Installing and configuring PHP in (Open)Solaris
As most of you know the OpenSolaris Web Stack project has been busy integrating various open source components into OpenSolaris. An important part of the effort is also to make these components ready to use with minimal effort on your part.
Recently I was asked what does it take to set up PHP using the packages delivered by the Web Stack project. The good news is that the answer is: it just works! In the interest of highlighting how straightfoward it has now become to use PHP with OpenSolaris, here's the steps:
I'm using a machine with a freshly installed build 81 (http://opensolaris.org/os/downloads/) . You could do the same on build 79b (better known as the upcoming SXDE and Indiana preview 2).
Here are the steps to configure and enable PHP:
# svcadm enable apache22
And that brings me to the end of this HOWTO ;-)
To make sure it works, let's try the ever-popular phpinfo() page:
# cd /var/apache2/2.2/htdocs/ # cat > test.php <?php phpinfo(); ?> ^D #
Go access http://localhost/test.php on that machine to confirm the PHP page is working.
So yes, all you need to do is enable the apache22 service. If the PHP module is installed, everything works out of the box without any manual configuration (you may want to tweak the config to your needs, of course, but that's optional).
The key packages that make this work are SUNWapch22m-php5 and SUNWapch22r-php5 (which in turn depend on both PHP and Apache 2.2 packages) as these deliver mod_php and associated default configuration.
Final note: there is a bug (6653084) present in build 81 (it is not in SXDE or Indiana preview 2). If you are on b81 or b82, apache22 won't initially start, you'll need to work around the bug by renaming ssl.conf to something else:
# cd /etc/apache2/2.2/conf.d # ls ssl.conf ssl.conf # mv ssl.conf ssl.conf.sample
Posted at 12:53PM Jan 30, 2008 by jyri in WebStack |
Thursday December 20, 2007
Web Server 7.0u2
Fresh off the presses!
Web Server 7.0u2 is now available as of this afternoon! Update now!
This release includes support for the pkcs11-bypass that I've talked about before. You can see the release notes for more.
Posted at 11:11PM Dec 20, 2007 by jyri in WebServer | Comments[2]
Thursday December 06, 2007
That bit about performance...
Does web server performance matter?
Beyond performance, does efficient performance matter?
If yes, you may want to check this out!
Posted at 10:43PM Dec 06, 2007 by jyri in WebServer |
Tuesday December 04, 2007
Unconsolidating
I've mentioned in the past some of the complexities introduced by the consolidation model as we attempt to make open source components available for OpenSolaris.
Tonight I'll take a shot at listing some of the major difficulties with the SFW consolidation model when applied to the goal of making a broad set of packages available for OpenSolaris - something which will be critical as we move towards IPS.
Looking at Debian unstable I see there are close to 27000 packages available today. Looking at SFW I see about 100 components which produce 158 packages. As I mentioned before, that's not a complete comparison since other consolidations also deliver various open source components, so the total available for OpenSolaris is a good bit higher.
To succinctly state a goal, here's what I want to see:
% pkg status -a | wc -l 27000
So... how do we get there?
Or, to the topic of this entry, why can't we get there via SFW consolidation model?
1. SFW source/build model cannot scale
SFW today is one single source repository ( browse it here or download the tarball here). A built tree takes about 7.5GB. On a V2100z (dual Opteron) the build takes about 3 hours. To reiterate, this is about 100 components producing 158 packages. Let's say we succeed and end up with something on the order of 20000 packages (~126x)? That's close to a terabyte for each build tree and the build would take about 16 days ;-)
Those numbers are clearly beyond silly... it can't be done. I could just stop here since this reason alone guarantees that the current SFW consolidation model cannot be used going forward for too long. Admittedly getting to 20000 packages will take time so we can ignore this problem for a little while, but it is better to start planning now instead of waiting until a build takes more than one working day. BTW it only takes about 420 packages for the build to take a full working day (8 hours). Even if 20000 packages might be a lofty far away goal (but one I believe we must achieve), 420 packages is right around the corner.
Requirement: Individual components must be able to build & deliver without checking out or building the rest of the package universe.
2. Centralized breakage
If there is a bug in Ruby build why should the PHP development team be prevented from making progress because they can't build either? (Not to pick on Ruby which works great ;-). The single tree/single build nature of SFW means that if the build is busted for any reason, it's broken for everyone. Historically it hasn't been a consolidation that moves very fast - a few dozen components, most of which don't change at all, only a few changes trickling in each month. In that environment, the single point of global failure is quite manageable.
Once again, fast forward to thousands or tens of thousands of components being maintaned by hundreds of dispersed teams. Conservatively let's say each package changes (version updates or any bug fixes) only twice a year. With 20000 packages, that's already over a hundred changes every day of the year! It's pretty much guaranteed that the build will be broken nearly all the time (but it'll take you 16 days to find out why and by then close to 2000 additional changes have gone in!)
Requirement: Bugs & build problems in some components cannot stop progress on entirely unrelated components.
3. Serialization of efforts
During the past four months (duration of the Web Stack project) we had less than half a dozen teams actively putting back changes into SFW. Even with such a tiny numbers, there have been times when one team is held up because the consolidation gate is waiting on some other completely unrelated putback.
As before, this will not work when the level of activity goes up. Once you go from half a dozen teams and a handful of checkins a week to hundreds of teams performing a hundred checkins a day, any serialization will cause the queue of pending changes to quickly grow out of control.
Requirement: Independent teams need to be able to check in code and deliver packages without contending on a single synchronization point.
4. Release early, release ... eventually?
We're all familiar with the phrase release early, release often. The consolidation model used by SFW is the opposite of this idea. The consolidation model assumes that there is one development team with the resources to do all the development and all the testing necessary to polish the component to perfection. Only after perfection has been reached does it get integrated into the consolidation, after which no further work is usually needed (at least until new features are requested). Admittedly there elegance in this approach. Unfortunately it also doesn't really apply to the task of packaging third party open source components, where the community feedback loop embodied by "release early, release often" is a vital part of the cycle.
Here is a concrete example taken from my team's initial PHP integration:
In early August we had PHP packages suitable for installation and experimentation. While there were known problems they wouldn't have prevented using the packages for early testing. Unfortunately we had no convenient way of publishing them because the consolidation model needs the fully polished final version to be done before putback. So we missed out from any potential feedback on these early packages which would've been quite valuable.
By mid-September the work was complete and the packages were ready. By now I had set up an internal IPS repository for distributing the finished work inside Sun. But as we have no external IPS repository to publish into, we still had no convenient way to truly publish the completed work. Again we're missing out on all potential community feedback.
In mid-October all the processes have completed and we make the checkin into SFW consolidation. Unfortunately checking the code in doesn't really make it available unless you have access to the SFW gate and are willing to build it yourself. It's not until 2 weeks later that the packages show up when snv_b76 .iso files become available. And after that, it's roughly another week before b76 is available for
download. So it's only in early November that the general community - all of you - has easy access to readily installable packages, even though preliminary packages suitable for testing were available to us three full months earlier. The worst part is that by then it is so late in the release cycle for SXDE that even if we get suggestions or bug reports, there is little to no time to act on them. It is the kind of feedback that would've been most useful back in August.
Requirement: Ability for each component team to independently release early, release often.
5. Schedule synchronization
Another difficulty introduced by the consolidation model is the synchronization of schedules of all components. Since it is a single source tree built at once (every 2 weeks) it requires all putbacks to be synchronized around that beat. This is a lesser issue but even then, even at the current slow pace of change, there's been some inefficiencies introduced by this. Even minor inefficiencies can become a problem once scaled up to thousands of packages and/or contributors so it is worth keeping in mind. Particularly for packages being maintained by community members who don't necessarily work to Sun's schedule.
Requirement: Same as #3: Independent teams need to be able to check in code and deliver packages without contending on a single synchronization point.
Well, it's easy to see that the existing consolidation model doesn't quite work for the goal of massively scaling up the number of packages available for OpenSolaris. But then, what's next? In my next article I'll explore some thoughts on ways we might move forward.
Posted at 08:42AM Dec 04, 2007 by jyri in WebStack |
Sunday December 02, 2007
PHP connectors are in OpenSolaris
The headline is vague, but it refers to the final round of PHP enhancements discussed over in the OpenSolaris Web Stack project for delivery into the Nevada build (b79) that will become SXDE 1/08 later (certainly there is a lot more more work and enhancements in the future ... but this is the final set that was in the roadmap for SXDE 1/08).
This is an assorted mix of new features (read more in the ARC case but the most important ones were the mod_php package for Apache support and the MySQL extension support following the integration of MySQL just a few days ago into b79. Now mod_php (SUNWapch22[mr]-php5), MySQL extension (SUNWphp524-mysql[-root]) and Postgres extension (SUNWphp524-pgsql[-root]) are delivered as separate packages which pull in their appropriate dependencies. This means that you no longer have to install all of these apps (Apache, MySQL & Postgres) only because you want to run PHP, conversely, if you install the extension you want (say, MySQL extension) it'll make sure the correct apps are in place so it can be used.
This is a first step towards refactoring all these packages along functional lines which will be quite important as we move to a la carte package installation with IPS. There's a long history of such overly broad dependencies in Solaris packages which hasn't been much of an issue while installing the whole boatload off of one DVD. As we move to IPS I want to make sure it's possible to install sensible subsets of functionality without having to bring in too many unwanted dependencies. If you have any preferences along these lines, drop by the Web Stack project and let us know!
Posted at 11:57PM Dec 02, 2007 by jyri in WebStack |
Wednesday November 28, 2007
MySQL 5 is in OpenSolaris
Between Thanksgiving break and so many other things going on, I'd missed mentioning that MySQL 5 is now also in OpenSolaris (as of last week) and should show up whenever b79 makes it's slow winding way through the system to your friendly opensolaris.org download site.
With that all the major building blocks of S A M P are now in! Of course, there's more to come. For instance, now that MySQL is in we'll need to deliver the PHP extension to talk to it, along with other connector pieces. For details of what's to come, check out the OpenSolaris Web Stack project...
Posted at 11:56PM Nov 28, 2007 by jyri in WebStack |
Tuesday November 27, 2007
Consolidation Cathedral
I'm constantly reminded how the SFW consolidation model is such a prototypical cathedral... components arrive through careful orchestration, highly serialized, all funneled into the same repository. All are built together in lockstep, delivered biweekly in one big ball up the chain, combined with other consolidations and eventually much later made available to the masses from the balcony.
(Too bad I can't draw or there would be a good opportunity for a fun cartoon here ;-)
Which is not to say this cathedral model is necessarily wrong. For some things (say, ON) it does seem quite applicable. On the other hand, the goal of SFW is to make available an inherently decentralized set of components: assorted open source packages delivered [mostly] as-is from their respective communities into OpenSolaris.
So there lies our challenge.. how to break down the cathedral walls of SFW and convert it into the bazaar model?
Posted at 11:58PM Nov 27, 2007 by jyri in WebStack | Comments[2]
Monday November 19, 2007
Apache (2.2.6) is in OpenSolaris
And now, this week it is the turn for Apache... Of course, Apache httpd has been in Solaris for a long time, but the update to 2.2.6 that's been discussed for a while now on the Web Stack project list is now in OpenSolaris. It'll show up once build 78 becomes available... Thanks to Arvind and Seema for all the hard work!
Rahul is still working on some additional extensions which will hopefully be available some time later.
Posted at 08:46PM Nov 19, 2007 by jyri in WebStack | Comments[2]
Subscribe!