|
Wednesday September 19, 2007
ZFS backups to USB mass storage ? 
If something were available that, on plugging a USB disk into an OpenSolaris laptop, would automatically snapshot any filesystems you cared about, and would send incremental ZFS send-streams to that disk (incremental, based on the last stored snapshots to that disk), would this be useful to anyone?
I've been playing around with HAL and D-Bus over the last few days, and I've nearly got something that works - thanks for the tip Darren! You specify the ZFS filesystems you want saved, along with the volume names of the USB disks which should be used as backing stores (these are assumed to be pcfs for now) and everything just works.
As with my previous SMF explorations, this will be an easy-to-configure SMF service, and will be totally automatic once you've set it up.
You could configure multiple backup services across different USB devices, say watching for a particular 1gb memory stick for frequent small incremental backups of your development workspaces, as well as say, a 700gb external disk, for less frequent full larger backups of different filesystems.
The theory here, is that the best backups are the ones you don't have to think about - plugging a disk into your laptop, and having the right stuff happen seems like a pretty easy solution to me.
This stuff still needs polish, but depending on feedback, I'll spend more/less time on finishing this up, but it was one of my ideas for the "ZFS to the Max" talk in October so I hope to have something concrete by then.
(2007-09-19 09:38:30.0) Permalink Comments [6]
Yes, this would be very useful. When we evaluated the ideal options for the backup of laptops in the lab, two main ideas emerged. One is pretty much exactly what you are describing. The other one is that whenever a connection to the internet is detected (instead of a connection to a specific usb disk), a backup is started (maybe unless one was already done very recently).
One thing that might not be completely trivial is dealing with the case where the connection to the USB device / the internet is removed before the transfer is complete (but it should not be too hard either).
Sadly, most laptops run linux these days and even the Macs won't have smf, but this would still be useful to some.
Posted by Marc on September 19, 2007 at 06:11 PM IST #
Okay, I've got this working now. Still needs a bit of work to determine whether there's enough space on the USB device to save the snapshot - and in order to do that, I need to store the send stream on the local machine first, then copy it over if it fits.
That sucks a bit, but as a first pass, it might be okay, and assuming there's lots of space on the host machine seems like a reasonable assumption for now, but feedback welcome.
For the default instance, the user needs to set an SMF property to specify the default backup volume name, then set a particular zfs user-property on any datasets they want backed up. After that, it all justs work when that device is inserted.
Additional instances can be created, with different USB volumes, and named zfs datasets - the idea being to make life easier for the administrator, being able to turn on and off backups at will.
I'll post more when it's ready for more eyes.
Posted by Tim Foster on September 20, 2007 at 05:34 PM IST #
An integrated solution to backup to an USB HDD would be a great feature for notebook and desktop users.
I'm looking forward to read more bout this ...
Posted by Arthur on September 21, 2007 at 12:14 AM IST #
What would really be useful is a hosted service that let me do that instead.
Posted by Steven on September 21, 2007 at 07:43 PM IST #
With a little work, I think we could also create hooks triggered from the Network Automagic project ( http://opensolaris.org/os/project/nwam/ ) such that when you connect your laptop to a network, a similar backup of your ZFS filesystems would be performed.
I'd need to look into the details of whether the current phase of NWAM allows this (I believe it does). Work would needed on the remote end to configure your backup server, but it's definitely possible, using ZFS to send a backup stream over an ssh connection to a remote server.
Indeed, the ZFS Automatic Snapshot service already allows for remote backups, as of version 0.6 - more detail at http://blogs.sun.com/timf/entry/zfs_automatic_snapshots_now_with
Posted by Tim Foster on September 22, 2007 at 03:13 PM IST #
Updated post here:
http://blogs.sun.com/timf/entry/zfs_automatic_backup_0_1
Posted by Tim Foster on October 01, 2007 at 05:03 PM IST #