Friday Apr 04, 2008
Friday Apr 04, 2008
The flashback is still alive even weeks after: the day before my presentation at FIRST Technical
Colloquium in Prague I brought my 2 years old laptop with the work-in-progress slides to the office.
Since I wanted to finish the slides in the evening a live-upgrade
process was fired off on the laptop to get fresh Nevada
version. (of course, to show off during the presentation ;))
LU
is very I/O intensive process and the red Ferrari notebooks tend to get _very_ hot. In the afternoon I
noticed that the process failed. To my astonishment, the I/O operations started to fail.
After couple of reboots (and zpool status / fmadm faulty commands)
it was obvious that the disk cannot be trusted anymore. I was able to rescue some data
from the ZFS pool which was spanning the biggest slice of the internal disk but not all data.
(ZFS is not willing to get corrupted data out.) My slides were lost as well as other data.
After some time I stumbled upon James Gosling's
blog entry about ZFS mirroring on laptop.
This get me started (or more precisely I was astonished and wondered how is it possible
that this idea escaped me because at that time ZFS had been in Nevada for a long time)
and I have discovered several similar and more
in-depth
blog entries about the topic.
After some experiments with borrowed USB disk it was time to make it reality
on a new laptop.
The process was a multi-step one:
Part Tag Flag Cylinders Size Blocks 0 root wm 3 - 1277 9.77GB (1275/0/0) 20482875 1 unassigned wm 1278 - 2552 9.77GB (1275/0/0) 20482875 2 backup wm 0 - 19442 148.94GB (19443/0/0) 312351795 3 swap wu 2553 - 3124 4.38GB (572/0/0) 9189180 4 unassigned wu 0 0 (0/0/0) 0 5 unassigned wu 0 0 (0/0/0) 0 6 unassigned wu 0 0 (0/0/0) 0 7 home wm 3125 - 19442 125.00GB (16318/0/0) 262148670 8 boot wu 0 - 0 7.84MB (1/0/0) 16065 9 alternates wu 1 - 2 15.69MB (2/0/0) 32130
AVAILABLE DISK SELECTIONS:
0. c0d0
/pci@0,0/pci-ide@12/ide@0/cmdk@0,0
1. c5t0d0
/pci@0,0/pci1025,10a@13,2/storage@4/disk@0,0
# zpool create -f data mirror c0d0s7 c5t0d0For a while I have struggled with finding a name for the pool (everybody seems either to stick to the 'tank' name or come up with some double-cool-stylish name which I wanted to avoid because of the likely degradation of the excitement from that name) but then chosen the ordinary data (it's what it is, after all).
root:moose:/data# mkfile 10g /data/test & [1] 10933 root:moose:/data# zpool status pool: data state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 mirror ONLINE 0 0 0 c0d0s7 ONLINE 0 0 0 c5t0d0 ONLINE 0 0 0 errors: No known data errorsIt survived it without a hitch (okay, I had to wait for the zpool command to complete a little bit longer due to the still ongoing I/O but that was it) and resynced the contents automatically after the USB disk was reconnected:
root:moose:/data# zpool status pool: data state: DEGRADED status: One or more devices are faulted in response to persistent errors. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the faulted device, or use 'zpool clear' to mark the device repaired. scrub: resilver in progress for 0h0m, 3.22% done, 0h5m to go config: NAME STATE READ WRITE CKSUM data DEGRADED 0 0 0 mirror DEGRADED 0 0 0 c0d0s7 ONLINE 0 0 0 c5t0d0 FAULTED 0 0 0 too many errors errors: No known data errorsAlso, with heavy I/O it is needed to mark the zpool as clear after the resilver completes via zpool clear data because the USB drive is marked as faulty. Normally this will not happen (unless the drive really failed) because I will be connecting and disconnecting the drive only when powering on or shutting down the laptop, respectively.
# chmod A+user:vk:add_subdirectory:fd:allow /data # zfs allow -s @major_perms clone,create,destroy,mount,snapshot data # zfs allow -s @major_perms send,receive,share,rename,rollback,promote data # zfs allow -s @major_props copies,compression,quota,reservation data # zfs allow -s @major_props snapdir,sharenfs data # zfs allow vk @major_perms,@major_props dataAll of the commands had to be done under root.
$ zfs create data/vk
dataset properties comment +-------------------+--------------------------------+------------------------+ | data/vk/Documents | copies=2 | presentations | | data/vk/Saved | compression=on exec=off | stuff from web | | data/vk/DVDs | compression=gzip-8 exec=off | Nevada ISOs for LU | | data/vk/CRs | compression=on copies=2 | precious source code ! | +-------------------+--------------------------------+------------------------+So the commands will be:
$ zfs create -o copies=2 data/vk/Documents $ zfs create -o compression=gzip-3 -o exec=off data/vk/Saved ...
However, this is not the end of it but just beginning. There are many things to make the setup even better, to name a few:
With all the above the data should be safe from disk failure (after all disks are often called "spinning rust" so they are going to fail sooner or later) and also the event of loss of both laptop and USB disk.
Lastly, a philosophical thought: One of my colleagues considers hardware as a necessary (and very faulty) layer which is only needed to make it possible to express the ideas in software. This might seem extreme but come to think of it. ZFS is special in this sense - being a software which provides that bridge, it's core idea to isolate the hardware faults.
tags:
laptop
mirror
notebook
opensolaris
raid
solaris
usb
zfs
Linkage:
Technorati cosmos