Monday May 12, 2008
I have had a busy couple of months. After wrapping up work on Solaris 8 Containers (my teammate Steve ran the Solaris 9 Containers effort), I turned my attention to helping the Image Packaging team (rogue's gallery) with their efforts to get OpenSolaris 2008.05 out the door.
Among other things, I have been working hard to provide a basic level of zones functionality for OpenSolaris 2008.05. I wish I could have gotten more done, but today I want to cover what does and does not work. I want to be clear that Zones support in OpenSolaris 2008.05 and beyond will evolve substantially. To start, here's an example of configuring a zone on 2008.05:
# zonecfg -z donutshop donutshop: No such zone configured Use 'create' to begin configuring a new zone. zonecfg:donutshop> create zonecfg:donutshop> set zonepath=/zones/donutshop zonecfg:donutshop> add net zonecfg:donutshop:net> set physical=e1000g0 zonecfg:donutshop:net> set address=129.146.228.5/23 zonecfg:donutshop:net> end zonecfg:donutshop> add capped-cpu zonecfg:donutshop:capped-cpu> set ncpus=1.5 zonecfg:donutshop:capped-cpu> end zonecfg:donutshop> commit zonecfg:donutshop> exit # zoneadm list -vc ID NAME STATUS PATH BRAND IP 0 global running / native shared - donutshop configured /zones/donutshop ipkg shared
If you're familiar with deploying zones, you can see that there is a lot which is familiar here. But you can also see that donutshop isn't, as you would normally expect, using the native brand. Here we're using the ipkg brand. The reason is that commands like zoneadm and zonecfg have some special behaviors for native zones which presume that you're using a SystemV Packaging based OS. In the future, we'll make native less magical, and the zones you install will be branded native as you would expect. Jerry is actually working on that right now. Note also that I used the relatively new CPU Caps resource management feature to put some resource limits on the zone-- it's easy to do!. Now let's install the zone:
# zoneadm -z donutshop install
A ZFS file system has been created for this zone.
Image: Preparing at /zones/donutshop/root ... done.
Catalog: Retrieving from http://pkg.opensolaris.org:80/ ... done.
Installing: (output follows)
DOWNLOAD PKGS FILES XFER (MB)
Completed 49/49 7634/7634 206.85/206.85
PHASE ACTIONS
Install Phase 12602/12602
Note: Man pages can be obtained by installing SUNWman
Postinstall: Copying SMF seed repository ... done.
Postinstall: Working around http://defect.opensolaris.org/bz/show_bug.cgi?id=681
Postinstall: Working around http://defect.opensolaris.org/bz/show_bug.cgi?id=741
Done: Installation completed in 208.535 seconds.
Next Steps: Boot the zone, then log into the zone console
(zlogin -C) to complete the configuration process
There are a couple of things to notice, both in the configuration
and in the install:
- Non-global zones are not sparse, for now
- Zones are said to be sparse if /usr, /lib,
/platform, /sbin and optionally /opt are
looped back, read-only, from the global zone. This allows a substantial
disk space savings in the traditional zones model (which is that the zones
have the same software installed as the global zone).
Whether we will ultimately choose to implement sparse zones, or not, is an open question. I plan to bring this question to the Zones community, and to some key customers, in the near future.
- Zones are installed from a network repository
- Unlike with traditional zones, which are sourced by copying bits from the global
zone, here we simply spool the contents from the network repository.
The upside is that this was easy to implement; the downside is that
you must be connected to the network to deploy a zone. Getting the bits
from the global zone is still desirable, but we don't have that implemented
yet.
By default, zones are installed using the system's preferred authority (use pkg authority to see what that is set to). The preferred authority is the propagated into the zone. If you want to override that, you can specify a different repository using the new -a argument to zoneadm install:
# zoneadm -z donutshop install -a ipkg=http://ipkg.eng:80
- Non-global zones are small
-
Traditionally, zones are installed with all of the same software
that the global zone contains. In the case of "whole root" zones
(the opposite of sparse), this means that non-global zones are about
the same size as global zones-- easily at least a gigabyte in size.
Since we're not supporting sparse zones, I decided to pare down the install as much as I could, within reason: the default zone installation is just 206MB, and has a decent set of basic tools. But you have to add other stuff you might need. And we can even do more: some package refactoring should yield another 30-40MB of savings, as packagings like Tcl and Tk should not be needed by default. For example, Tk (5MB) gets dragged in as a dependency of python (the packaging system is written in python); Tcl (another 5MB) is dragged in by Tk. Tk then pulls in parts of X11. Smallness yields speed: when connected to a fast package repository server, I can install a zone in just 24 seconds!.
I'm really curious to know what reaction people will have to such minimalist environments. What do you think?
Once you start thinking about such small environments, some new concerns surface: vim (which in 2008.05 we're using as our vi implementation) is 17MB, or almost 9% of the disk space used by the zone!
- Non-global zones are independent of the global zone
- Because ipkg zones are branded, they exist independently of the global zone. This means that if you do an image-update of the global zone, you'll also need to update each of your zones, and ensure that they are kept in sync. For now this is a manual process-- in the future we'll make it less so.
- ZFS support notes
- OpenSolaris 2008.05 makes extensive use of ZFS, and enforces ZFS
as the root filesystem. Additional filesystems are created for
/export, /export/home and /opt. Non-global zones don't yet follow this convention.
Additionally, I have sometimes seen our auto-zfs file system
creation fail to work (you can see it working properly in the example above). We haven't
yet tracked down that problem-- my suspicion is that there is a bad interaction
with the 2008.05 filesystem layout's use of ZFS legacy mounts.
As a result of this (and for other reasons too, probably), zones don't participate in the boot-environment subsystem. This means that you won't get an automatic snapshot when you image-update your zone or install packages. That means no automatic rollback for zones. Again, this is something we will endeavor to fix.
- Beware of bug 6684810
-
You may see a message like the following when you boot your zone:
zoneadm: zone 'donutshop': Unable to set route for interface lo0 to éÞùÞ$ zoneadm: zone 'donutshop':
This is a known bug (6684810); fortunately the message is harmless.
In the next month, I hope to: take a vacation, launch a discussion with our community about sparse root zones, and to make a solid plan for the overall support of zones on OpenSolaris. I've got a lot to do, but that's easily balanced by the fact that I've been having a blast working on this project...
(2008-05-12 06:16:16.0) Permalink Comments [26]
Tags: opensolaris solaris zones
Trackback: http://blogs.sun.com/dp/entry/a_field_guide_to_zones
I'll save most of my comments for when this makes it to zones-discuss. Until then, I'll offer these thoughts:
- Proper dependencies will be a huge help to create zones that are not such pigs.
- To enable the automatic building of zones that have the right software, the zone configuration should have resources that specify IPS repositories to use.
- Follow-on to above, zone configuration should have resources that specify packages to install.
- I like the idea of a branded zone. If there is a way to specify which ABI the kernel expects, the zone could have a resource that knows what the ABI is and allow the branded zone to be attached to anything that supports that ABI. This would prevent minor differences in the global zones from various boxes from causing a non-global zone to become incompatible with them during attach/detach operations.
- As I've mentioned before - spend the time on ZFS dedupe rather than sparse zones.
To illustrate my point configuration resources:
# zonecfg -z donutshop
donutshop: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:donutshop> create
zonecfg:donutshop> set zonepath=/zones/donutshop
zonecfg:donutshop> add net
zonecfg:donutshop:net> set physical=e1000g0
zonecfg:donutshop:net> set address=129.146.228.5/23
zonecfg:donutshop:net> end
zonecfg:donutshop> add capped-cpu
zonecfg:donutshop:capped-cpu> set ncpus=1.5
zonecfg:donutshop:capped-cpu> end
zonecfg:donutshop> add ipsrepo
zonecfg:donutshop:ipsrepo> add ipsrepo
zonecfg:donutshop:ipsrepo> set authority=com.myco
zonecfg:donutshop:ipsrepo> set url=http://pkg.myco.com/
zonecfg:donutshop:ipsrepo> end
zonecfg:donutshop> add ipsrepo
zonecfg:donutshop:ipsrepo> set authority=org.blastwave
zonecfg:donutshop:ipsrepo> set url=http://pkg.blastwave.org:10000/
zonecfg:donutshop:ipsrepo> end
zonecfg:donutshop> add pkg
zonecfg:donutshop:pkg> set authority=org.myco
zonecfg:donutshop:pkg> set name=feature/sampstack
zonecfg:donutshop:pkg> set version=2008.05
zonecfg:donutshop:pkg> end
zonecfg:donutshop> commit
zonecfg:donutshop> exit
Notice that the only package added is pkg://org.myco/fature/sampstack-2008.05. This, however, is my way of controlling what is in sampstack via my private repository rather than having to maintain it on hundreds of machines. It in turn pulls in the bits that it needs from the default repository, pkg.blastwave.org, and pkg.myco.com.
These resources are only useful during installation. However, absent some other mechanism to perform a fully automated build of zones they are really quite important.
Posted by Mike Gerdts on May 12, 2008 at 09:34 AM PDT #
Are you seriously suggesting removing sparse zone functionality permanently?
That's a horrible idea, why on earth would you do that?
At the very least would we still be able to build zones off ZFS snapshots so as to avoid a 5 gig BIND install?
Posted by JohnS on May 12, 2008 at 04:04 PM PDT #
@John-- I'm not sure what you mean by 5 gig BIND install? Could you elaborate? Because we have so much flexibility in the new packaging system, it seems like one could simply install a minimal zone (see my comments about "Non-global zones are small" above), then add SUNWbind (2.91 Mb). But maybe I'm not understanding your point...
At a higher level, we're saying that we want to take a look at this problem. Sparse zones are great, but also cause a lot of complexity; one possibility would be refactor the problem as you suggest-- using a "parent" zone and a bunch of derivative "child" zones. Or, we could look at developing a compositing file system. Please make sure you're on zones-discuss@opensolaris.org, as we'll be talking this through over there in the next 4-6 weeks.
Posted by 192.18.43.225 on May 12, 2008 at 05:08 PM PDT #
(oops-- that last comment was from me, dp).
Posted by Dan Price on May 12, 2008 at 05:08 PM PDT #
By "5 gig bind install" I mean this. At the moment I have BIND walled off in a sparse zone... Really only /etc exists inside the zone. Getting rid of sparse zones would involve an entire system instance ( SUNWall ) = about 5GB getting copied over for no real reason.
I thought the whole point of moving to a pre/postinstall-less packaging system was because zones made it difficult?
Posted by JohnS on May 12, 2008 at 09:20 PM PDT #
@JohnS: In OpenSolaris 2008.05, with Image Packaging, I can deploy a zone which contains just the very core OS, plus BIND, and that zone will occupy a total of about 220MB of disk space.
The point is that the old constraint of "a whole root zone starts with the same set of packages as the global zone" seems to not really match what people want to do with their systems.
We often hear from folks who really want to minimize as much as possible, for one reason or another. It seems that Image Packaging provides the ability to do so, and so zones can be "whole root" but still fairly compact.
Again, I'm going to write all of this up so we can talk about it in our community. Nothing has been decided yet.
Posted by Dan Price on May 13, 2008 at 02:02 AM PDT #
@dp & JohnS: Another important point is that IPS will make it much easier to not have a 5 GB global zone too. If the purpose of your global zone is to do enough that zones can run on a headless server, I bet that it would be rather easy to get this into way less than 500 MB. Today, getting that functionality is rather difficult.
Hmmm... maybe this suggests a good project to start in the Sysadmin community. Feature packages (http://mgerdts.blogspot.com/2008/03/solaris-wish-list-feature-based-meta.html) could be created to handle all of the dependencies for common scenarios such as a global zone to host zones, a SAMP zone, a DNS server zone, an IPS repository zone, etc.
Posted by Mike Gerdts on May 13, 2008 at 05:21 AM PDT #
I'm absolutely gobsmacked that you'd remove sparse root zones. You mention that people would like minimal zones and yet you remove sparse root zones?
"Non global zones are independent of the global zone"
Adding a dependancy of a network connection on being able to build a zone is just plain nuts. Let me clarify that: _JUST_ _PLAIN_ _NUTS_! Have you actually considered how zones are used?
If I install sun studio on the global zone, I can then build sparse root zones so that all sparse zones can have access to cc tools. That means several developers with several zones have almost no additional footprint. If we do it the opensolaris way, not only does sun studio have to be installed in each zone, it HAS to be downloaded EACH TIME!!!
Posted by Ronan on May 13, 2008 at 09:13 AM PDT #
@Ronan: This is a conversation. Nothing has been decided yet. I'm trying to be frank and open here so _SHOUTING_ _IS_ _NOT_ _APPRECIATED_. I'm listening to you and other who are providing feedback. However, I think there are a range of choices here.
As for the dependency on a network connection: that's a limitation of what we have in 2008.05 today: that was the easiest thing to do, and it was what we could get done in the time allotted. We'll be working to fix this, going forward.
As for the Studio install case, thanks for raising that, it's good input to the discussion.
Posted by Dan Price on May 13, 2008 at 01:21 PM PDT #
Well, with your example of a stripped down root install, that still leaves a 200 meg footprint for what should take up no more than half a dozen MB ( for logs & config files ) if it's created very sparse. Less if you set up a loghost.
Really, the zone that I run BIND in should need one config file and one folder of zone ( in the DNS sense ) files. grand total of less than 100kb.
Keeps things separate logically. If sparse zones are eliminated I'd stop using them & just run my apps in the global context so as to not waste disk space
Posted by JohnS on May 13, 2008 at 02:50 PM PDT #
@Ronan: One thing that IPS brings is the ability to use the packaging system to manage more than boot environments. According to http://opensolaris.org/os/project/indiana/files/GetStart.pdf:
[begin quote]
The image-create command sets up a relationship between the image and a server that can be accessed to add packages to the image.
Images can be of three types:
* Full images, capable of providing a complete system.
* Partial images, which are linked to a full image (parent image), but do not provide a complete system on their own.
* User images, which contain only relocatable packages.
[end quote]
I really hope that we can get to the point where the OS image (be it a global zone or a non-global zone) contains only the bits required for the OS. Applications (e.g. Sun Studio, OpenOffice, etc. should be relatively independent of the OS) and should be an image (or images) of their own. I think that Sun Studio is a perfect example of something that would be best as a User Image. That image could be shared (e.g. lofs) by multiple zones or even multiple hosts at different OS revisions. Then when you need to patch or otherwise update the image, you do it once for all hosts and zones.
Posted by Mike Gerdts on May 14, 2008 at 06:11 AM PDT #
It seems to me that in the new world of zfs root the most intuitive way to generate a new zone is to zfs clone the original install. If a zfs snapshot was taken of the original boot environment that could be used to clone from. In this world boot environments and zones are essentially equivalent (zones being BE's with restrictions). Zone creation would be incredibly fast, give the equivalent of a whole root zone and take up no space.
Posted by Mark Phalan on May 16, 2008 at 02:02 AM PDT #
The biggest real win for sparse root zones has nothing to do with the disk space but everything to do with sharing of cached binaries. Otherwise you end up with a separate copy of libc etc in memory for every zone because they all come from different ZFS filesystems.
Posted by Darren Moffat on May 16, 2008 at 11:59 AM PDT #
Can i have iscsi initiators running inside a non global zone?
Posted by arun on May 19, 2008 at 03:48 AM PDT #
@Mark: That is true, right up until the OS images (in the zones, and in the global zone) begin to diverge. We do support zone creation using ZFS cloning today (on Nevada), and I fully expect to continue to support that.
@Darren: That's a matter I am still looking into. I'm trying to get some quantification of how much memory (say, as a percentage of total system memory) the sharing buys you on modern systems. It may be that with even modest systems rapidly reaching 4GB of memory (or more), and with zones running a more stripped down image (hence less unique bits of text), that text sharing will be less of a win than it was in the past. But we don't yet know.
@Arun: Not today, no-- but you can certainly hook up to your iSCSI volumes from the global zone and then place them into the zone, using the "add device" command in zonecfg(1m). If you need control over initiators inside the zone, please cruise over to http://bugs.opensolaris.org and file a feature request. Thanks!
Posted by Dan Price on May 19, 2008 at 11:45 AM PDT #
You've got to be kidding me. I setup a zone on OpenSolaris 200805 thinking that I could surf the web in a safe environment.
So I ftp'd Firefox 3.0b5 from their ftp server and it wouldn't run. I noticed that Gnome Libs was missing so I pkg install Gnome Libs and then it mentions that fonts aren't installed.
So cd /; du -sh; and it shows as only ~270 MB's are installed.
So from your solution I need to download about 1 GB from the 'net just to run a browser?
Unreal.
alan
Posted by Alan Pae on May 19, 2008 at 06:48 PM PDT #
Folks, did you ready Dan's original blog post and his subsequent replies? Nowhere did he say that what's shipped in 2008.05 reflects what will be available in future releases. Rather, it's an initial cut of zones support using the new packaging system. Based on constructive input, I have no doubt that zones support in the next long term supported version of Solaris will be superior to that of Solaris 10.
@Ronan, have you tried to share the Sun Studio tools using a lofs mount? That still should work fine in 2008.05.
Posted by David Comay on May 20, 2008 at 06:41 PM PDT #
Let me start again. I like IPS zones. A zone independent of the global zone can be useful. Having a zone with different versions of core packages is useful.
However I would suggest:
1) Make IPS zones an additional type of branded zone, but leave the default zone type the same as in plain vanilla solaris.
Zones are a killer app for Solaris 10. Its on the main bullet point list for Solaris 10. Developers are using it regularly and happily. Making a huge change to a major subsystem is a big deal, and leaves developers who depend on functionality irritated :) (Especially if they'd just downloaded opensolaris and were about to install it and looking forward to some zone work...).
A lot of developers have invested a lot of time in the current zone system and are now dependent on its interface.
A lot of developers find the sparse zone system to be exactly what they need. And with an initial outlay of maybe 80Mb is _very_ lightweight.
2) Make the initial repository for an IPS zone to be the global zone. You've already got the packages, simply reuse them. The IPS zone will still run/be upgradable independently of the global zone, but you don't necessarily _need_ to access the network.
Following on from that, options a user might then want:
a) Default is to take from global zone.
b) Only get packages from the network, like 2008.05 functionality.
c) Check the network repository but only download _newer_ packages. Allowing a lighter load on the network.
Posted by Ronan on May 21, 2008 at 10:00 AM PDT #
Folks, did you ready Dan's original blog post and his subsequent replies? Nowhere did he say that what's shipped in 2008.05 reflects what will be available in future releases. Rather, it's an initial cut of zones support using the new packaging system. Based on constructive input, I have no doubt that zones support in the next long term supported version of Solaris will be superior to that of Solaris 10.
===
Actually, yeah we did. I also have no doubt that things will get better in the long run however in the near term I think most of us assumed that it would at least maintain the status quo.
I have not been able to get Firefox to run even after doing a:
pkg install SUNWfirefox
which moves ~350 MB's of files from your system to mine. The funny thing is after doing this, Opera installs and runs fine.
We know things will get there but for now its growing pains.
alan
Posted by Alan Pae on May 21, 2008 at 10:45 AM PDT #
@Alan: In my own defense, we're lucky to have support for zones *at all* in 2008.05; the other option was to simply disable them completely. The basic level of support currently present happened because I worked really hard to make it happen. For reasons beyond my control, 2008.05 did not have as part of its ship criteria any support for zones, at all, AFAIK.
Additionally, because of the complex interaction between whatever packaging system you have and zones, it's going to take a significant engineering effort to get Image Packaging to where we need to go. Just as it did when we integrated System V packaging and zones.
As for your problem with Firefox, I'll take a look when I am back from vacation. It's very likely to be a missing dependency problem in one or more packages.
@Ronan: We're planning to do suggestion #2. Just not there yet; it's not quite as simple as you suggest due to the way Image Packaging works (just as it wasn't trivial with System V packaging). As with System V packaging, we'll need to invent a way to save off the factory-fresh versions of editable files, or something, so that we can use those instead of the edited ones during install.
Posted by Dan Price on May 21, 2008 at 11:49 PM PDT #
@Alan: In my own defense, we're lucky to have support for zones *at all* in 2008.05; the other option was to simply disable them completely.
===
Yea, I got that from the top of this page. :-)
I moved another 250 Mb's of files using pkg install and now I almost have Firefox 3.0rc1 working. The file selector for where to save downloads isn't working but truss will take care of that. Have a good vacation. I know this is going through growing pains.
alan
Posted by Alan Pae on May 22, 2008 at 12:51 PM PDT #
One more thing.
How do I get sound to work? I added in Adobe's flash player and it shows the video but there is no audio output.
From zonecfg info:
device:
match: /dev/sound/*
System, Administration, Device Driver Utility lists my audio as using audio810 driver, native sound support.
Thanks,
alan
Posted by Alan Pae on May 22, 2008 at 01:03 PM PDT #
@Alan: I think that I read elswhere you need to add /dev/audio.
Posted by Mike Gerdts on May 22, 2008 at 01:19 PM PDT #
@Mike: Probably. We should probably invent a template for "desktop zone" which just makes all of this "just work"...
Other than Trusted Extensions, I wasn't really aware of folks wanting to use zones for desktop type applications; something new to think about!
Posted by Dan Price on May 23, 2008 at 01:35 PM PDT #
HI Dan,
wow, seems easy to set up zones - and quick, 24 seconds!!! (though I'm sure my 1/2 mbit connection will take a little longer =).
My question is, if you do not mind (as i have NRTM yet), can I run solaris 10 in a opensolaris/2008.05 zone.
thanks
Posted by phil on July 16, 2008 at 03:53 AM PDT #
Hi Phil,
from: open solaris discussion forum ===========================
Not currently. Zones are currently limited to "ips" branded zones.
I believe this is in progress and should be available in the relatively near future.
================================================
Dan, will this (possibly) be in 08.11?
Thanks
Posted by 86.140.39.127 on July 16, 2008 at 04:05 AM PDT #