The Open Road

ZFS and File System State Management

ZFS, available from Solaris 10 update 2 on, is one amazing piece of technology. Bloggers like Drew get what I get about ZFS, which is that once you grok its features and start combining them, you can do some weird and wonderful things.

State management of training machines is something I've been trying to do well for about 15 years now. I've been through a lot of technologies like Norton ghost, scripting, jumpstarting, and OS virtualized images and while they all work, they also have limitations that often make me have to settle for less than what I really want to do.

So I was playing around with a VMware Solaris 10 Update 2 a couple of months ago that had /z, a big ol' ZFS file system, about 5 GB on it, and a whole root zone completely encapsulated within the ZFS file system. Here's what I was able to do with it:

1. First, I took a ZFS snapshot called z@machinestartstate.
2. I booted and logged into the whole root zone.
3. I set the zone up to have the start state for one of the classes for which I am responsible. Let's call it class 1.
4. I took another ZFS snapshot called z@class1startstate.
5. I saved the deltas between the two snapshots: zfs send -i z@machinestartstate z@class1startstate > machinestartstate.class1startstate.
6. I did Lab 1.
7. I took another ZFS snapshot called z@class1lab1endstate.
8. I saved the deltas between the two snapshots: zfs send -i z@class1startstate z@class1lab1endstate > class1startstate.z@class1lab1endstate.
9. I did Lab 2.
10. I took another ZFS snapshot called z@class1lab2endstate.
11. I saved the deltas between the two snapshots: zfs send -i z@class1lab1endstate z@class1lab2endstate > class1lab1endstate.class1lab2endstate.
12. I followed the same steps for Labs 3 through 10, and then did the same things for 5 other classes I am responsible for.

Now look at the possibilities I have. As long as I can get back to my original virtual machine image, I can play the delta files back to any state I want:

• zfs rollback -r z@machinestartstate always takes me back to the original virtual machine state.
• zfs recv < machinestartstate.class1startstate
restores the start state for class 1.
• zfs recv < class1startstate.class1lab1endstate
restores the end state for Lab 1 of class 1.

and so forth. In subsequent blog entries, I'll be writing about how to create a Solaris 10 VMware virtual machine for this setup, and provide workarounds for some issues I found while getting this working.

Posted at 10:16AM Oct 09, 2007 by dgolds in Solaris  |  Comments[1]

Geekfest in San Francisco

My friends Rick, Brandon and I rode "Paradise Loop" yesterday. This is an outlandishly beautiful bike tour that takes you around the Tiburon Peninsula in Marin County, north of San Francisco. The David-modified version of this loop includes a detour to Larkspur before the final climb of the day up Camino Alto. The reason for the detour is a trip to Caffe Rulli, a place that makes you feel as though you've left California and entered Siena.

So we were enjoying our cappuccinos, cake, and gelato after our nice 27 mile ride, when Rick and Brandon started arguing whether I was really a geek.

Brandon: Well, he actually enjoys talking about virtualization.

Rick: Yeah, but he doesn't play video games.

David: Yeah, and I don't like Star Trek either, but I do like The Simpsons. Especially that episode where Brandine tells Custis, "You're the best husband and son I've ever had."

Brandon: He's too athletic to be a geek. He's this kind of weird combination of bike rider / techie.

David: Athletic? I was always the one picked last at sports.

Brandon and Rick, almost in unison: So was I...

Rick: Well, he's kind of geeky.

I brought that conversation home with me after the ride, because my desk looked like this:

There I was with my geek stuff all spread out, trying to figure out how to get my new laptop hard drive to have the same files on it as the old hard drive.

My favorite new toy is the Vantec SATA / IDE to USB 2.0 Adapter. Our group's lab services guy, Chris, had suggested it and sent me his as a loaner. This thing is pure genius. First you pull the old drive out of the laptop. Then you stick the new drive in and format it. I did this by doing a quick and dirty SuSE 10.2 installation. During installation, I partitioned the drive as I wanted (I wanted bigger /home and /vm partitions) but made sure to keep the size of the root partition the same size as the root partition on the old drive. I instinctively thought it would help - my inner geek was speaking to me loud and clear on that one.

The installation was simple and took about half an hour. Aren't modern Linux distros wonderful? Next, I cut a Fedora 7 rescue CD. If you've never used one of these things, it's worth knowing about. The rescue CD lets you boot from the CD, then mounts the root directory from your hard drive (if it can access it) under /mnt/sysimage so you can fix things like /etc/fstab or /boot/grub/menu.lst. Turns out this rescue CD is smart enough to know how to mount a USB device, too.

So I attached the old drive (which at this point was out of the laptop) to the Vantec adapter, stuck its USB cable in my laptop's USB port, and booted my system from the rescue CD. It worked like a champ - after boot up finished, I could see all the partitions on my new drive (which by this time I had installed in the laptop), and could mount partitions on the old drive.

Then it was time to move the data over to the new drive. I only have 3 partitions on the drive, and I got the data over using the following technique:

• dd if=/dev/sdb2 of=/dev/sda2 to copy the root partition
• mkdir /mountpoint to create a mount point
• mount /dev/sdb6 /mountpoint to mount the /home directory from the old drive
• cd /mountpoint
• find . -xdev | cpio -padm /mnt/sysimage/home to copy all the files in /home from the old drive to the new drive
• umount /mountpoint
• mount /dev/sdb7 /mountpoint to mount the /vm directory from the old drive
• find . -xdev | cpio -padm /mnt/sysimage/vm to copy all the files in /vm from the old drive to the new drive
• Modified the /mnt/sysimage/etc/fstab and /mnt/sysimage/boot/grub/menu.lst files so that system would be able to boot and all the file systems would be mounted correctly
• grub-install /dev/sda to write the boot loader to the new drive

I rebooted the laptop, this time from the hard drive, and it came up looking exactly like it did with the old drive, except with bigger /home and /vm partitions just as I wanted.

I did go down one wrong path, which was to copy the files in the root partition to the new drive with the find | cpio command instead of dd. After doing this, the system booted but none of the passwords worked so I couldn't log in as any user. I don't know why I had problems, perhaps there was some cruft left over from the quick and dirty SuSE 10.2 install caused problems. But when I switched dd to copy the root partition, the problem went away.

Posted at 06:42AM Oct 08, 2007 by dgolds in Other  |  Comments[0]