Tim Foster's Weblog

I'm just about done with this release of the ZFS Automatic Snapshot SMF service and have just pushed some changes to the mercurial repository on opensolaris.org.

This is a pretty important release, in terms of fixing stuff that's been bugging me about the service since I initially released it. But inevitably, with lots of change comes the possibility of lots of bugs - so, I was hoping to get some feedback on how it's looking before it gets officially released.

So, if you're feeling brave (that means don't use this in production yet!) fire up your favourite source code management system (which is hg, right?) and access the mostly-untested ZFS Automatic Snapshot 0.11 Early Access release via:

hg clone ssh://anon@hg.opensolaris.org/hg/jds/zfs-snapshot

I'm working with Niall & Erwann in the Desktop group here, who have been tasked with DSK-5, to get ZFS Automatic Snapshots on the desktop, and so far, it looks like my code will be providing the back-end service (obviously as well as ZFS :-) so some of the changes are things that make most sense when running this on a desktop or laptop machine.

With that in mind, I've not made any changes to my bundled GUI, since it'll be going away real soon now. However, I've done my best to ensure that there's always ways of turning off the small-system-focused bits, and the service remains backwards-compatible with earlier manifests.

So what's going to be new in 0.11 ? Well having seen Nils write up his changes in that form (more on that later), I thought I'd have a go at writing a Changelog too - so here's the annotated Changelog entry so far for 0.11:

0.11

Add RBAC support

• the service now runs under a zfssnap role
• service start/stop logs stay under /var/svc/log
• other logs saved to /export/home/zfssnap (and syslog) [ yes, this sucks a bit - better solutions welcome? ]

Add a 'zfs/interval' property value 'none' which doesn't use cron

Add a cache of svcprops to the method script (good idea Nils!)

Add a com.sun:auto-snapshot user property used by all instances, com.sun:auto-snapshot:$LABEL takes precedence

Remove the seconds field of the snapshot name - it's not needed (good idea Håkan!)

Changed the way // works with recursive snapshots - ignore snapshot-children, and instead automatically determine when we can take recursive snapshots based on which datasets have the zfs user properties

Set avoidscrub to false by default (6343667 was fixed in in nv_94)

Bugfix from Dan (thanks!)- Volumes are datasets too

Automatically snapshot everything by setting com.sun:auto-snapshot=true on startup. (this gets done on all top level datasets - an existing property set to false on the top level dataset overrides this)

Check for missed snapshots on startup

Clean up shell style a bit

Clean up preremove script (I need to make these scripts redundant before we move to IPS, I know)

Write this Changelog

In terms of user-visibility, the most obvious changes are running under RBAC, and taking snapshots of all filesystems by default - I realise the latter could be controversial, but you can turn it off if you don't like it. I'm also pretty happy with the changes to the "//" schedule - we now ignore "zfs/snapshot-children" for this particular case, and instead use the list of filesystems marked as "com.sun:auto-snapshot=true" to work out which filesystems we can take recursive snapshots of, and which we have to take indivdual snapshots of. This makes a big difference on large systems.

One thing that's missing from this release, is Nils Goroll's suggested changes about improving the way the system performs scheduling - more details here. I feel that moving away from cron would result in less familiarity in what the service does: if cron is the problem, certainly one solution is running away from it, but wouldn't it be cool to get cron's shortcomings fixed instead? Yeah, one of those "ample-free time" problems.

So, without further ado, there's full documentation in the README - enjoy, and please let me know if you see anything weird - there's still time to fix it before 2008.11 (and yes, all this despite my day-job being super hectic right now! xVM Server is getting 99% of my time at the moment, so I definitely expect bugs this early access release of zfs-auto-snapshots!)

Just back from 2 weeks vacation (had a great time, very relaxing, but I'm currently scaling mountains of laundry and email simultaneously) and I'm just starting into what's going to be my last week working in the ZFS Test group.

Before I left for our holidays in Spain, I found out that I'd been offered a job I'd been interviewing for - a position in the xVM development group. So I'm not going to be working in OpenSolaris QE anymore - rather than trying to find bugs in bits of OpenSolaris, I'm now going to be actively trying to avoid putting them in to begin with!

Specifically, the xVM guys wanted someone to help out with their gate-keeping work, freeing up John a bit more to work on other stuff. The timing here is good I think - we use Mercurial to manage the xVM gates, and as you know there's a project ongoing to move the rest of ON development over from Teamware to Mercurial (the SCM Migration project) so my getting involved here seems like a pretty exciting place to start. I'm expecting to be swooshing along a fairly steep learning curve quite soon - wish me luck!

For now, I'm going to be spending the rest of the week catching up on email, and making sure there's no loose ends with the ZFS test suite development work that I'd been helping with. As soon as I've found my feet in the new group, I'll let you know how things are going.

ZFS is still a rather exciting project to be involved with, so I'll try to keep up to date with what's happening (and intend to continue maintaining my automatic snapshot utilities, of course) - I've learnt a great deal during my time on Solaris QE, and will definitely miss working with the fantastic engineers who built ZFS - you guys rock!

Oh, and no - I know nothing about gin joints either!

Simon Breden wrote to me the other day, pointing to a nice set of articles he'd written about how he setup a home file server using ZFS. It's a good write up on everything from choosing the hardware to thinking about disk layout and exploring some of the ZFS commands needed to create the server.

This is another case study that shows that ZFS isn't just for enterprises, it can be used quite nicely at home as well - well worth a read and a good introduction to ZFS if you're just getting started.

I've got a new version of the ZFS Automatic Snapshot SMF Service finished.

This release contains two bugfixes, one pointed out by Reid Spencer, and the other from Breandan Dezendorf - thanks for the bugs guys, much appreciated!

There's also a small new feature in this release, suggested on the zfs-discuss mailing list by Eric Kustarz. That is, that the service should avoid taking snapshots when zpool resilvering or scrubbing is happening.

I'm hoping this feature is only a temporary requirement - but 6343667 has more detail.

The implementation of this is pretty basic, but it works fine - there's a new property in the SMF manifest "zfs/avoidscrub", which is set to true by default. When executing the crontab entry, if this property is set to true, we check to see that the pool backing the dataset we're about to snapshot is currently mid-scrub or resilver. If it is, we report a message to syslog, and skip the current dataset for this invocation of the cron job.

I've updated the README and you can download these changes as zfs-auto-snapshot-0.10.tar.gz.

Hope this of use to people out there. As always comments and bug reports would be most welcome!

I'm releasing what might be the last set of ZFS Automatic Snapshot & Backup tools on this blog! Don't worry - the services aren't going away, but some folks in the Desktop group are starting to looking more carefully at this stuff, so they might move to opensolaris.org as a project somewhere. More on that later.

I've got two new packages available now - the main changes in each are more GUI integration, just some simple zenity tools to allow users to configure the services without needing the command line, which will appear in the GNOME menu under "Administration" after install - you may need to pkill gnome-panel or logout to see the change.

I'm not happy with the rights model for these GUI tools (they run using gksu to root), but hopefully they'll be replaced soon. I've also gone to more trouble to package these services properly, so installation/removal is much easier. More details below.

ZFS Automatic Snapshot Service 0.9

Apart from the GUI code being new, I've a few extra bits here:

• Packaging - we now have a Makefile and build a package, called TIMFauto-snapshot for now. Suggestions on better package names welcome. These are just SVR4 packages - I haven't had bandwidth to play with IPS yet, so beware the postinstall/preremove scripts!


• Adding some pre-configured SMF instances of common use-cases, eg. daily, hourly, weekly and monthly snapshot schedules, so users don't have to worry about creating a manifest for these.


• The inclusion of a special "fs-name" value, "//" which makes the service look at the ZFS user property "com.sun:auto-snapshot:<label>" so that users can mark filesystems to be included in a given snapshot schedule without needing to touch the SMF properties. The pre-configured instances above use this.


• Changing the timeout for the start and stop method to be infinite - the 10 second timout was occasionally causing the service to give up trying to start when lots of instances were on the system. Because of the way I'm adding cron entries, I need to serialize the startup for these instances, so that they can take longer than 10 seconds to start, depending on the number of instances you have. This isn't ideal, so better suggestions are welcome. I think this may fix the problem you were seeing Siegfried, but I still haven't been able to reproduce it exactly.


• A user had commented that having ":" characters in the snapshot names makes them inaccessible from samba clients - so I've added a $SEP variable to the method script to allow administrators change this. I was in two minds as to whether enough people would want this that I should add it as a new SMF property instead, but for now, if you need this, just edit line 64 of the method script.

Check out the updated README for more information.

Not much has changed here - the GUI is the main addition, along with putting the service in it's own package, the same caveats as per the auto snapshot service apply here - this package is currently called TIMFauto-backup.

Check out the updated README for more information.

Now, with the news out of the way, I'd like to ask for help. Given that both of these services are likely to get more attention in the future (the folks putting together Indiana think these would be really handy to have in the distro, which I'm really thrilled about) I want to see if we can get some more eyes on the code.

If there's anyone out there who'd like to offer code-reviews of either of these services, I'd be really interested in hearing from you. I use these on a regular basis myself, and I know there's quite a few folks out there who use them too, so they do work as intended (or at least have done so far, Murphy's law dictates that these versions I'm releasing today will kick your cat and burn your house down!) - however, as far as I know they've never been code-reviewed, from either a high-level architectural point of view, or a nitty gritty "your indenting sucks" perspective.

Would these be candidates for the Desktop consolidation, or some other consolidation targeting Indiana, and perhaps even Solaris some day ? Certainly the automatic backup service is currently quite desktop oriented, but a scheduled snapshot service sounds like something we'd want on servers too - ON perhaps ? (I'm not sure how to initiate that decision-making process either way)

So please, all comments would be welcome - I can take it, and if it means scrapping these implementations altogether and going back to the drawing board, then this would be a really good time to find that out!

Now - you've read this far (thanks) - here's the software. In each tarball, there's a prebuilt "proto" directory containing the package (which is architecture neutral). To install, as root simply do:

 # cd <install dir>/proto
 # pgkadd -d <name of package>

• zfs-auto-backup-0.2.tar.gz
• zfs-auto-snapshot-0.9.tar.gz

Enjoy!

I'm heading out to California tomorrow morning - need to get to Dublin airport for a 6:45 flight to Heathrow, followed by a 11ish connection to San Francisco. This means getting up at 4am - I can take it. By Thursday at the latest (hopefully Wed afternoon) I'll be wandering around the halls of MPK17, so say "hi" if you see me!

I'm very much looking forward to the weekend's OpenSolaris Summit and the subsequent Sun conference (no, it's not CEC - though I wonder how I could wrangle myself into getting there next year, sounds like fun!) Along with meeting lots of OpenSolaris people, who I've only known from email so far and have never met in person (_ ok, with a few exceptions _), one of my tasks for the weekend, is moderating the "ZFS to the max" discussions on Sunday, which started here, wandered happily along, and so far have made it as far as hereish.

With Indiana, we've got a huge opportunity to push ZFS as the default filesystem on a new operating system distribution and show the world how things could be - I really hope we'll have the courage to do so, and take best advantage of the capabilities it offers. That said, I'm all ears about what other weird and wonderful things ZFS could let us do on a developer-oriented distribution of OpenSolaris. Let's really go for it and see what happens!

As a side note, I'm also going to be at the mercy of the sat-nav system I got recently - here's hoping those maps are up to date. If you don't see me in Santa Cruz, then you know why :-)

I finished the first version of the ZFS Automatic Backup Service that I mentioned a while back.

You can download the zfs-automatic-backup-0.1.tar.gz package, which contains this README.zfs-auto-backup.txt documentation.

The intent here, is to be able to automatically backup selected ZFS filesystems on your machine whenever a named USB volume is plugged into your system. The target audience here is developers & users of small systems who want a small safeguard against their machine being lost or damaged - a little more peace of mind in the face of a heartless world.

It's pretty easy to setup - just copy the files to the right place, import the SMF manifest, and configure the name of the volume you want to backup to. Then use "zfs set com.sun:auto-backup=true <name of dataset>" to mark filesystems for backup.

On plugging the USB disk into the machine, the service takes a snapshot of the filesystems marked for backup, checks for the presence of an existing backup on the storage media, and kicks off either a full or incremental zfs send to the USB disk, depending on whether it finds a previous snapshot saved to the disk or whether you've explicitly asked for full backups all the time. More info in the README.

Here's a few screenshots showing what appears on your desktop whenever you plug the configured USB disk into your machine (at the start, and end of the backup):

The GUI notification is a bit hairy, an SMF service running as root writing something to the desktop session of a logged-in user, so apologies for that in advance (no, there's no multi-seat support, yes this is a terrible hack). Walking the line between "annoying pop-ups" and "not enough information" is difficult, I admit - so let me know what you think. There's a problem with text wrapping in the notification box - haven't yet investigated that.

I've tested this on small numbers of small filesystems, but it should scale to multiple-gigabytes of backups, assuming there's no bugs in my code (I was testing all this on a USB 1.0 card reader, so waiting around for gigabytes of data to copy wasn't an option :-)

Once again, many thanks to pjd for the "ZFS Man" image. All comments, and especially bug reports would be great!

If something were available that, on plugging a USB disk into an OpenSolaris laptop, would automatically snapshot any filesystems you cared about, and would send incremental ZFS send-streams to that disk (incremental, based on the last stored snapshots to that disk), would this be useful to anyone?

I've been playing around with HAL and D-Bus over the last few days, and I've nearly got something that works - thanks for the tip Darren! You specify the ZFS filesystems you want saved, along with the volume names of the USB disks which should be used as backing stores (these are assumed to be pcfs for now) and everything just works.

As with my previous SMF explorations, this will be an easy-to-configure SMF service, and will be totally automatic once you've set it up.

You could configure multiple backup services across different USB devices, say watching for a particular 1gb memory stick for frequent small incremental backups of your development workspaces, as well as say, a 700gb external disk, for less frequent full larger backups of different filesystems.

The theory here, is that the best backups are the ones you don't have to think about - plugging a disk into your laptop, and having the right stuff happen seems like a pretty easy solution to me.

This stuff still needs polish, but depending on feedback, I'll spend more/less time on finishing this up, but it was one of my ideas for the "ZFS to the Max" talk in October so I hope to have something concrete by then.

It's been a long time since I've talked about my actual day job - I used to write about it all the time when I worked in the G11n group, but these days, I tend to just write about ZFS, and not necessarily my contributions to it from a testing perspective.

Disclaimer: as a QE engineer, it's really difficult to know how to write about software testing - just doing so acknowledges that our software has bugs. So, can we just take the fact that software has bugs as a given, and move on?

Hint: your software has bugs too, so look out!

I started working in ZFS-test back in October 2005 and it's kept me really busy trying to keep on top of the performance improvements, feature additions and bug fixes that have been rolling in on a fairly constant basis from the developers since then. Life is far from dull. What's more, with every bug comes a little more knowledge about how ZFS works - I'm still learning a new thing every day, and that's good, I like learning new stuff.

Last week, I was doing what every good test developer should be doing - expanding our test coverage, and that's what I wanted to write about here.

As you can see from our released the ZFS Test Suite we've got a lot of coverage of the functional areas of ZFS, but there's always places where we can improve test coverage.

I was concentrating last week (or trying to, more on that later) on increasing the level of coverage for functional testing of ZFS commands when run as non-root users. With the advent of ZFS delegated administration, along with the specific tests for that feature, we wanted to double-check that we're doing the right for non-root users of ZFS utilities in a general sense - ensuring that they don't have any more permissions than they should, so it made sense to build-out our user tests. I'm thankful to say, the ZFS utilities are looking pretty solid here. Yeah!

It goes without saying, that just because we're only now writing those tests, it doesn't mean that non-root use of ZFS utilities has never been tested, remember these are just the formal automated-test cases: for every test case we write, there's a great many people using and running ZFS in ways which we haven't formally tested - in fact, just last week, I had a bug rear it's head on my desktop machine which our test suite hadn't caught! :-(

As a test development team, every time this happens to us, we feel a bit inadequate - it's a fact of life that we can't catch all the bugs of course, but that doesn't make us feel any better.

This particular bug was an OS panic - which, I should say first, we haven't seen on a production version of ZFS. Needless to say, this is somewhat distracting when it happens to your desktop (Windows blue screens, anyone?) - in fact, I think it might be the first time I've seen ZFS code that made it back into the Nevada gate actually panic my own machine. Ouch.

As every good developer should, I'm running the latest bits on my desktop, hoping to a) avoid the Quality Death Spiral, and b) expand coverage of ZFS testing.

As I said, our test suite covers a lot of functionality but it has bounds to the extent of what it can do to setup particular scenarios. In this case, the question was, how do you reproduce a particular bug that I've only been able to reproduce so far on a well-exercised, 30gb pool with filesystems (most created back in May 2006 - lots of churn since then) containing my test development & OS development workspaces, home-directory backups, lots of BFU archives from developers, iPod contents, usr/local & blastwave bits, etc. amounting to 26 filesystems and ~800 snapshots which are taken and deleted at constant 10-minute, daily, weekly and monthly intervals ?

Even if we could generate the right set of circumstances to make this bug appear, but it took 4 days of constant filesystem activity just to setup the test case, we wouldn't be able to roll it into the test suite - as it'd impact the runs of the rest of the suite, so getting short, easy to reproduce tests are what we're after. (though if such a test existed, we could run it periodically)

So far, we're still trying to get a test case for this bug, but this also demonstrates why formal test cases aren't a substitute for running your own software on your own machines, they're a compliment to interacting with your code in as many ways as you possibly can. At Sun, as a company we have the "Sun on Sun" program, where we're encouraged to run our own software in daily usage - indeed, you'll see many folks on blogs.sun.com talking about this, for example, Chris' series Good morning build...". Likewise, a lot of the engineering folks (myself included) run with remote home directories on a machine called "jurassic" over in Menlo Park, which also runs ZFS and has also shaken a few more bugs out of the filesystem.

Finding and catching bugs before users do is the reason why we test developers exist, and we take a belt & braces approach to testing. Even if you're not a test developer, you can still contribute to the effort - download your OpenSolaris distribution of choice, give it a spin and if you find anything amiss with ZFS, let us know via zfs-discuss or bugs.opensolaris.org. (the same goes for other technologies in OpenSolaris of course)

Your daily use of ZFS contributes to the quality of the filesystem, and that's a good thing for everyone, so kick those tyres!

Great stuff, finally I can talk about my day-job! Jim posted details about the ZFS Test Suite and where you can download it.

The ZFS test suite covers a wide range of ZFS functionality, not stress or performance tests yet. With over 400 test cases (and a lot more variations within) we think it's pretty good at testing ZFS functionality, and as the ZFS developers can hopefully can attest to, it does find bugs from time to time :-)

We're adding more tests all the time, and the hope to be able to do all of our future test development in the open, rather than using a periodic source-drop, of which this is the first. Still, small steps for now - we'll get there eventually.

If you're interested in seeing what exactly we're looking at, here's the current list of the extracted assertion statements from the ZFS test suite generated by a little bit of awk this afternoon. A pretty interesting read!

The README contains more information about using the suite, let us know if you've any questions about it?

As Mike points out, at the moment there's 7 active developers on the test suite, but we'd love more help, so let us know if you can spot functional test areas we've missed, or whether you're interested in helping out?

Eric wrote a marvelous post about ZFS on laptops, which I think is not only an excellent position statement about the merits of ZFS running on things other than just multi-disk servers, but also a pretty good introduction to the capabilities of ZFS and how easy it is to administer.

I'm a fan of running ZFS on laptops & desktops as well - I've got 7 years worth of photos and 18gb of iTunes music backed up to a 160gb disk running ZFS at home. Knowing that my data is safe is pretty reassuring (though I really must buy a 2nd disk to make a mirror and thus sleep even more soundly, sorry Constantin I'll get one eventually :-) )

I do take occasional backups to DVD as well, but ZFS gives me the confidence that when I'm doing those backups, I'm not backing up bits that have been silently corrupted - ZFS will show bit rot as checksum errors if it encounters these, even it if can't fix them (in a non-redundant configuration). End-to-end checksums: another good reason to run ZFS on your laptop.

Anyway, nice post Eric!

As promised, I've now got an updated version of the old mountrootadm script that works with the new ZFS Bootable Datasets. It allows you to create multiple LiveUpgrade-like boot environments. Of course, it isn't a complete LiveUpgrade solution, but is enough for me. Tim also wrote about this sort of thing recently.

You can download zfs-bootadm.sh now.

Like mountrootadm, zfs-bootadm.sh allows a user to easily maintain several bootable datasets, creating, destroying, activating and listing them. It also updates the GRUB menu.lst file, so on boot, you get a menu of all bootable datasets.

In order to set it up, create a ZFS root system (this script may help) and do two things:

1. set the user-property "bootable:" on the root dataset, eg.
   zfs set bootable:=true pool/root_filesystem
2. Append the name of the current ZFS root filesystem to it's title in your your menu.lst file, which would be in /pool/boot/grub/menu.lst)

The user-defined property, bootable: is just used by the script to work out which datasets are being managed by it - it has nothing to do with the underlying ZFS boot functionality.

Once you've done that, you can start using the script:

 # ./zfs-bootadm.sh
Usage: zfs-bootadm.sh [command] <arguments>

where command is one of:
         create <name>
                Creates a new bootable dataset as a clone
                of the existing one.
         activate <name>
                Sets a bootable dataset as the next
                dataset to be booted from.
         destroy <name>
                Destroys a bootable dataset. This must not
                be the active dataset.
         list
                Lists the known bootable datasets.

# ./zfs-bootadm.sh list
onnv-gate:2007-03-30
test (current)
timmy
# ./zfs-bootadm.sh create my-test    
bootable dataset space/my-test has been created, and mounted on /tmp/bootable_dataset.101339
# ./zfs-bootadm.sh activate my-test
Currently booted from bootable dataset space/test
Creating ram disk for /tmp/bootable_dataset.101339
updating /tmp/bootable_dataset.101339/platform/i86pc/boot_archive...
 this may take a minute
updating /tmp/bootable_dataset.101339/platform/i86pc/amd64/boot_archive...
 this may take a minute
On next reboot, bootable dataset space/my-test will be activated.
# ./zfs-bootadm.sh list
my-test (active on next reboot)
onnv-gate:2007-03-30
test (current)
timmy

There's a few things I'm not entirely sure about - I don't like requiring the menu.lst titles to end with the name of the bootable dataset, but it makes the menu.lst file easier to manage. I'm not happy with the name of the script either - too much potential confusion with the real bootadm. Also, I've only tested this on one machine here, but it appears to work fine (yeah yeah, I'm guilty of it too sometimes)

I was half-thinking of using a "bootable:master" setting to specify a single dataset to use as a base for cloning new datasets - that way there wouldn't be the same cascading dependency between multiple bootable datasets (eg. pool/c cloned from pool/b@c, pool/b cloned from pool/a@b, meaning that destroying pool/a is awkward without promoting lots of clones). If you think this would be useful, let me know.

Thoughts/comments welcome.

For the last while, I've been helping with the testing of the recently announced ZFS Boot bits that Lin putback yesterday.

We've got the regression testing on these bits completed - these changes don't break existing ZFS functionality, and we've validated that the basic functionality of ZFS bootable datasets works as designed.

I'm now looking at some additional tests for these bits, trying to boot mirrors with missing/detached disks, that sort of thing. (This week, I brought up a Thumper with root on a 47-way mirrored pool! :-)

As with the ZFS Mountroot bits before, I thought that writing a script to automate the install of these bits would be pretty useful while we don't yet have full ZFS support in the installer. Here it is: zfs-actual-root-install.sh.

This is how you use it:

root@usuki[88] ./zfs-actual-root-install.sh --help
Usage : zfs-actual-root-install.sh [options to pass to zpool]
eg. ./zfs-actual-root-install.sh mirror c0t0d0s0 c0t0d1s0

You need to be running a fresh install of at least snv_50
 (with a BFU of Lin's zfsboot bits) for this to work.

Note also, you must supply a disk using slice notation: we need SMI
labels to boot, whereas "zpool create c0t0d0" would use EFI labels.
Only single disks, or mirrors are supported. No stripes or raidz please.

If you set the environment variable $ROOT_FS, we use that as the root
filesystem.

As mentioned above, ZFS root boot only works with SMI labeled disks - if you've ever given ZFS the entire disk before, it'll have put an EFI label on the disk, so you need to remove that using fdisk, then rewrite the label using format, or fmthard. Not too scary - here's me having just changed the disk type:

             Total disk size is 8924 cylinders
             Cylinder size is 16065 (512 byte) blocks

                                               Cylinders
      Partition   Status    Type          Start   End   Length    %
      =========   ======    ============  =====   ===   ======   ===
          1       Active    Solaris2          1  8923    8923    100


SELECT ONE OF THE FOLLOWING:
   1. Create a partition
   2. Specify the active partition
   3. Delete a partition
   4. Change between Solaris and Solaris2 Partition IDs
   5. Exit (update disk configuration and exit)
   6. Cancel (exit without updating disk configuration)
Enter Selection: 5


format> l
[0] SMI Label
[1] EFI Label
Specify Label type[1]: 0
Warning: This disk has an EFI label. Changing to SMI label will erase all
current partitions.
Continue? y
Auto configuration via format.dat[no]? 
Auto configuration via generic SCSI-2[no]?

Here's the script in action:

root@usuki[92] ./zfs-actual-root-install.sh mirror c2t0d0s0 c2t1d0s0
Updating vfstab on UFS root
Starting to copy data from UFS root to /zfsroot - this may take some time.
.
.
.
10576640 blocks
.
.
There's a copy of the old UFS root in /zfsroot/etc/vfstab.old-ufs-root
diffs are new vs. old :
6a7
> /dev/dsk/c0d0s0       /dev/rdsk/c0d0s0        /       ufs     1       no      -
12,13c13
< rootpool/rootfs - / zfs - no -
< /dev/dsk/c0d0s0 /dev/rdsk/c0d0s0 /ufsroot ufs - yes -
---
> rootpool/rootfs - /zfsroot zfs - yes -
Creating ram disk for /zfsroot
updating /zfsroot/platform/i86pc/amd64/boot_archive...this may take a minute
updating /zfsroot/platform/i86pc/boot_archive...this may take a minute
Installing grub on /dev/rdsk/c2t0d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 260 sectors starting at 50 (abs 16115)
Installing grub on /dev/rdsk/c2t1d0s0
stage1 written to partition 0 sector 0 (abs 16065)
stage2 written to partition 0, 260 sectors starting at 50 (abs 16115)

Okay, assuming we haven't broken anything, when you next reboot, you
should be able to select a grub menu entry for ZFS on root!
Remember to report anything suspicious via bugster or
zfs-discuss@opensolaris.org.
If your boot device has changed because of this, remember to change your
bios settings.  (you should now boot from /dev/dsk/c2t0d0s0 /dev/dsk/c2t1d0s0)

And finally, here's me booting with the new root:

# df -h / 
Filesystem             size   used  avail capacity  Mounted on
rootpool/rootfs         67G   4.6G    62G     7%    /
# zfs list
NAME              USED  AVAIL  REFER  MOUNTPOINT
rootpool         4.57G  62.4G    24K  /rootpool
rootpool/rootfs  4.57G  62.4G  4.57G  legacy
# zpool status -v
  pool: rootpool
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        rootpool      ONLINE       0     0     0
          mirror      ONLINE       0     0     0
            c2t0d0s0  ONLINE       0     0     0
            c2t1d0s0  ONLINE       0     0     0

errors: No known data errors
# 

You can install a root pool to a slice that isn't slice 0, but in that case, the script won't work this out, and will run installgrub to that slice - if that's the case, you should manually run the installgrub command to put the new ZFS-capable grub on whatever your boot device is.

One other thing to watch out for, is that if you're BFUing development archives, you might run into 6528202 - so take a copy of /boot/platform/i86pc/kernel/unix before you BFU! If you're happy to wait for a full install of nv_62 or later, then you don't need to worry about this step.

I've said it before, but having ZFS on your root filesystem is just completely awesome - being able to incrementally backup, snapshot and rollback your root filesystem really is amazingly useful. I wrote mountrootadm to help out even more. Of course, eventually I suspect LiveUpgrade will handle all this for you, but in the meantime this does the trick.

Let me know if you've any thoughts or comments about the script. Happy rumbling!

Update: Bart quite rightly pointed out to us that there's no need for all that mucking around with failsafe-boot in order to reconstruct the /dev and /devices filesystems. Much easier and faster is:

mkdir -p /zfs-root-tmp.$$
mount -F lofs -o nosub / /zfs-root-tmp.$$
(cd /zfs-root-tmp.$$; tar cvf - devices dev ) | (cd /zfsroot; tar xvf -)
umount /zfs-root-tmp.$$
rm -rf /zfs-root-tmp.$$

So I've updated the post above to change that, fixed the script and tested it - works just fine. Thanks Bart!

Update: Lin pointed out a typo in the create_dirs script where /tmp was being given the wrong permissions, so I've fixed that in this version of the script too.

Update: - we were wrong about it being a typo. Normal service resuming..

Here's a small update to the ZFS Automatic Snapshot SMF service first mentioned here. There's a few bugfixes in this release, and a small feature addition.

The first bugfix is the one that Dick kindly pointed out - sorry about that, all working now.

The second was a bit more subtle. Every once in a while, when starting my machine, I noticed that although the snapshot service instances were all listed as online, with no errors in their logs, nothing was being snapshotted - the entries that should have been added to the system crontab on starting the service weren't there.

It turns out that this was because I didn't have any form of locking around my accessing the crontabs, so the service instances were stepping on each others toes: in this case, SMF was being too damned efficient and was starting all my instances in parallel (as it should! :-) The locking I'm using now probably isn't the best - but let me know what you think. Seems to work fine.

I've also added a small feature - the addition of a "verbose" property for the instances, with the default setting being false. I have a recursive, rolling snapshot every 10 minutes on one of my filesystems, and I was getting tired of my system logs getting spammed all the time by my instance deleting older snapshots, so that's made life a bit nicer.

As always, let me know if you've any feedback - I'm always interested in ways I can improve this service.

The new version is now available in zfs-auto-snapshot-0.8.tar.gz and I've updated the README with these changes.

I've just finished a relatively minor update to the ZFS Automatic Snapshots SMF Service.

The main change here, was due to Bill complaining (nicely) that the service was a bit noisy when running from the cron script. Too right! Every time the service ran, it would spew the FMRI for the service - even when everything was working properly, it'd still print it, resulting in a noisier than necessary output.

(For some reason, I'm now quite sensitive to unexpected spewing, and can definitely sympathise about babies^H^H^H^H^H^Hdaemons that are noisier than they could be (Bananas had her 2 month immunisations today, and the poor wee soul is a bit under the weather as a result))

So as well as fixing that, I had another look at the way the method script is doing logging. Before, stdout from the script was being handled by cron, mailing the output to root. I've now changed all of the logging in the script, so that messages are now reported via logger(1) to syslogd(1M) : I'd still love to have the messages land in the appropriate SMF log for each instance, but since we're running the script from cron, not SMF, this is a bit tricky. With these changes, when running the service now, you'll messages in syslog like:

Nov 27 11:55:16 haiiro zfs-auto-snap: [ID 702911 daemon.notice] space/timf@zfs-auto-snap:frequent-2006-10-28-22:30:00 being destroyed as per retention policy.

- the service isn't too verbose, but let me know if you've any feedback. I've updated the README to mention this.

The new version is now available in zfs-auto-snapshot-0.7.tar.gz

Wow - Mika has a patch for my ZFS Automatic Snapshots service, to have it clean up old snapshots on the backup server.

We talked about this a bit via email, and I'm thinking I don't really want to incorporate the patch into this code. My thinking was two-fold - first, I'm not sure I'd really want clients futzing with the backup server: let the backup server admin decide what they want to do with the backups as they roll in.

Next, if you have a few hundred clients all running the ZFS Automatic Snapshots service, and you add diskspace to your backup server, you then perhaps need to change each client, in order to tell it not to destroy server-side backups any more.. Centralised administration, yeah?

Now, on the other hand, if we had a means of accessing a remote SMF repository, the client's zfs/backup-destroy-cmd could be retrieved from the backup server, properly allowing centralised administration of this feature.

Anyway, I think it's waay cool that Mika's thought about this and published his patch - he's got a two-server setup, where he wants backups from one machine to be copied over to the other machine, but also destroyed periodically. Not a backup server, so much as a "poor-man's cluster" (as Mika so eloquently put it) - so if you don't want to setup an actual cluster (for free!), then Mika's patch might be what you're after!

Update: Of course, I should also mention this thread, which talks about actual clustering using ZFS.. (as shipped, ZFS is not a clustered filesystem, don't plug a zfs pool into two hosts simultaneously, and expect it to work. more here and here)

This weekend, while playing about with OpenSolaris, I was looking to boot my laptop with some different bits. I didn't want to overwrite my current OS image, but unfortunately I didn't have the disk partitions available to create a new LiveUpgrade boot environment (nor the time to wait for it!)

Thankfully, I'd been running ZFS Mountroot pretty much since it was announced, so I was able to use ZFS snapshots and clones of my root filesystem, with a little editing in various etc/vfstab files to get me what I wanted.

But that got me thinking, it'd be neat if someone automated this - and besides, it wouldn't be a very complex script... Now, I know that ZFS Boot is probably just around the corner, but until LiveUpgrade groks ZFS, here's something to use in the meantime: mountrootadm.sh

Here's me using it :

(an initial one-time only step to setup it's configuration file)
# df -h /
Filesystem             size   used  avail capacity  Mounted on
tank/root_filesystem   9.2G   5.2G   4.0G    57%    /
#
# echo tank/root_filesystem > /ufsroot/opt/etc/mountroot.cfg

(now, onto it's usage..)
# ./mountrootadm
Usage: mountrootadm [command] {arguments}

where command is one of:
         create {name}
                Creates a new mountroot dataset as a clone
                of the existing one.
         activate {name}
                Sets a mountroot dataset as the next
                dataset to be booted from.
         destroy {name}
                Destroys a mountroot dataset. This must not
                be the active dataset.
         list
                Lists the known mountroot datasets.
# ./mountrootadm list
root_filesystem (current)

# ./mountrootadm create newroot
Mountroot dataset tank/newroot has been created, 
and mounted on /tmp/mountroot_dataset.1325

# ./mountrootadm list
root_filesystem (current)
newroot

# ./mountrootadm activate newroot
Currently booted from mountroot dataset tank/root_filesystem
On next reboot, mountroot dataset tank/newroot will be activated.

# ./mountrootadm list
root_filesystem (current)
newroot (active on next reboot)

All the script does, is create a named snapshot of the current root dataset, clone that snapshot to a new dataset, and put appropriate entries into /etc/vfstab and the corresponding /ufsroot/etc/vfstab (and update boot_archives appropriately)

Let me know if you find this useful, or if you run into any problems using it. As soon as ZFS Boot comes out, I'll write something similar for that environment, in case LiveUpgrade hasn't yet seen the light that is ZFS!

Update: Actually, on further testing, this script still needs some work. I'd forgotten about the mountroot bootadm hack, which always copies the current boot_archive into /ufsroot (overwriting the hard work we've done putting an alternate boot environment in there). So after activating a mountroot dataset, you need to shutdown the machine in such a way that the "bootadm update-all script isn't run. Now, "halt -n" will do this, but I'm not sure if that will have any harmful side effects...

Update:This might need some work in light of the recent ZFS bootable dataset putback. I'll hop to it!

I've finished writing the next round of features for my ZFS Automatic Snapshots SMF Service prototype. You can download this as zfs-auto-snapshot-0.6.tar.gz

The main new features in this release are:

• ZFS Send/Receive support
• Multiple schedules per filesystem

The send/receive support means that if you want it to, the service can send backups of each snapshot, either full streams or incremental streams, depending on how the service is configured. The service will also send snapshots of all child filesystems, if required, though without the send -r support in ZFS, this is a little unweildy a at the moment.

There's a SMF property which the user can set to the command which should recover the backup stream. Typically, this would be a "zfs receive", but there's no reason why you couldn't simply cat the output to a unique file on an NFS server. I've altered the bundled GUI to also ask for these new options when it's constructing a new manifest:

The multiple schedules per filesystem feature allows the user to assign an optional label to each snapshot schedule, allowing multiple schedules for the same dataset. For example, for a given filesystem you might choose to take monthly full backups, sent to a remote server (and backed up to tape as a flat file), but also daily incremental backups, perhaps via zfs send/receive to a different server.

The label is also useful to quickly tell which services are running for which filesystems. For example, here's the configuration on my desktop at the moment:

root@haiiro[236] svcs | grep zfs
online         Aug_31   svc:/system/filesystem/zfs/auto-snapshot:space-archive
online         Aug_31   svc:/system/filesystem/zfs/auto-snapshot:tank-root_filesystem
online         13:28:27 svc:/system/filesystem/zfs/auto-snapshot:space-timf
online         17:47:37 svc:/system/filesystem/zfs/auto-snapshot:default
online         18:00:02 svc:/system/filesystem/zfs/auto-snapshot:tank-new,backup
online         18:01:02 svc:/system/filesystem/zfs/auto-snapshot:tank-new,moreoften

I've updated the documentation and README for these new features, but let me know if anything's unclear.

Finally, I'm trying hard to do the right thing in the face of failure. The service will move to maintenance should a backup fail for any reason, and the cron job should be removed in that case. Also, I'm doing some basic locking, to see if zfs send commands are still running before attempting to send another backup stream from the same instance. Unfortunately, there doesn't seem to be an atomic way to set/get properties from SMF from what I can see, but feedback is welcome.

I hope you find this stuff useful, and if you run into problems, bug reports would be great!

ps. Chris is also doing some pretty snazzy stuff with ZFS snapshots - over on his blog : well worth checking out!

[ update here ]

The other day, Roger asked for an additional feature to be added to the ZFS Automatic Snapshots SMF service that I wrote. The feature was, the ability to configure multiple snapshot schedules per filesystem.

For example, I have a ZFS filesystem for which I've configured a snapshot service to take weekly snapshots, and keep them on the system for a few months. However, during the day, I also want to have the system take automatic hourly snapshots of that filesystem, but only keep perhaps snapshots from the last 6 hours.

"Easy!", I thought - and started writing the suppport for the feature. Along the way, I also added support for the new snapshot -r ZFS feature now available in Solaris Nevada, which really speeds things up.

However, I quickly ran into a wall (ouch) when thinking about this feature. At the moment, each instance of the snapshot service configures a different set of automatic snapshots. For example, here's what's on my desktop :

root@haiiro[59] svcs auto-snapshot
STATE          STIME    FMRI
disabled       16:49:03 svc:/system/filesystem/zfs/auto-snapshot:space-test
online         16:08:55 svc:/system/filesystem/zfs/auto-snapshot:space-archive
online         16:08:55 svc:/system/filesystem/zfs/auto-snapshot:space-timf
online         16:08:55 svc:/system/filesystem/zfs/auto-snapshot:tank-root_files ystem
online         16:49:03 svc:/system/filesystem/zfs/auto-snapshot:default

The FMRI of each instance nicely delimits the boundaries of each set of snapshots - the instance names don't have any special significance, other than to show what's happening on the system at a glance. All the useful information I need in order to take snapshots are actually stored in instance-properties.

However, when I started to think about multiple instances for the same filesystem, I kept running into namespace clashes (other than these)

I can't create instance names "svc:/system/filesystem/zfs/auto-snapshot:space-timf:hourly" or "svc:/system/filesystem/zfs/auto-snapshot:space-timf@hourly" or "svc:/system/filesystem/zfs/auto-snapshot:space-timf$hourly" - SMF doesn't allow those characters in instance names.

I think I'm looking to subdivide the instance somehow - I'd thought of individual property groups inside each instance, each one corresponding to a different schedule for that filesystem, but then the granularity of the service breaks down: I couldn't disable hourly snapshots while keeping the monthly snapshots enabled for a given filesystem. Maybe that's okay ?

I'll keep thinking about it. Along with that, I'm also still trying to find the best way for my cron job to log snapshot status into the SMF log - simply redirecting output from the cron script into each FMRI's corresponding restarter/logfile doesn't seem like the right thing to do. Perhaps more coffee will help - but otherwise, all thoughts are welcome!