Jim Laurent's Weblog

Update 2/28: Made some minor corrections.  Provided an English and high quality version of the German video.  Added a ZFS GUI screenshot and instructions.  Added a link to Constantin's ZFS and Virtual Box blog entry.

Being a Solaris (and Mac) propeller head and fueled by a few Coronas, I found it hard to refuse this challenge. For an advanced version of this, check out this YouTube video (high quality MP4 version) from my colleagues across the pond.  Here are the steps that I followed.

System under test:  MacBook Pro running MacOS 10.5.2, VMware Fusion 1.1.1 and Solaris 10 08/07.

 1. Enable USB device access per the VMware Fusion instructions: 

1	Open the virtual machine without powering it on. You cannot configure the USB controller while the virtual machine is powered on.

2	Choose Virtual Machine > Settings or click the Settings button in the toolbar to open the virtual machine Settings sheet.

3	Select + and Add USB controller.

4	Select one or both of the options:

•	Enable USB 2.0 Support

•	Automatically Connect USB Device

5	Click Apply.

2. Boot the Solaris VM. Login. Open a Solaris terminal window.  Assume root privileges.  Disable the Volume Management service volfs.  This prevents Solaris from automounting the removable disks. This stays in effect across reboots until you "enable" it.

    svcadm disable volfs 

3. Insert the USB hub with 3 sticks into the Mac's USB port

4. Fusion menus: Virtual Machine > USB > Connect ....  for each of the 3 USB devices.  This "grabs" them away from MacOS into Solaris control.

5. Find out the device names for the three USB disks:

# rmformat
Looking for devices...
     1. Logical Node: /dev/rdsk/c0t0d0p0
        Physical Node: /pci@0,0/pci-ide@7,1/ide@1/sd@0,0
        Connected Device: NECVMWar VMware IDE CDR10 1.00
        Device Type: DVD Reader/Writer
     2. Logical Node: /dev/rdsk/c2t0d0p0
        Physical Node: /pci@0,0/pci15ad,790@11/pci15ad,770@2/storage@1/disk@0,0
        Connected Device: CBM      Flash Disk       5.00
        Device Type: Removable
     3. Logical Node: /dev/rdsk/c3t0d0p0
        Physical Node: /pci@0,0/pci15ad,790@11/pci15ad,770@2/storage@2/disk@0,0
        Connected Device: USB      Flash Disk       1100
        Device Type: Removable
     4. Logical Node: /dev/rdsk/c4t0d0p0
        Physical Node: /pci@0,0/pci15ad,790@11/pci15ad,770@2/storage@3/disk@0,0
        Connected Device: CBM      Flash Disk       5.00
        Device Type: Removable

6.  Create a zpool using RAID Z on the three devices.

# zpool create usbdisk raidz c2t0d0p0 c3t0d0p0 c4t0d0p0
invalid vdev specification
use '-f' to override the following errors:
raidz contains devices of different sizes

# zpool create -f usbdisk raidz c2t0d0p0 c3t0d0p0 c4t0d0p0
# # zpool status

  pool: usbdisk
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        usbdisk       ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            c3t0d0p0  ONLINE       0     0     0
            c2t0d0p0  ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0

errors: No known data errors

# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
usbdisk                 360M     91K    360M     0%  ONLINE     -

7.  Now lets have some fun......

8. Create a 5 MB file

cd /usbdisk
mkfile 5m test
# ls -l
total 10245
-rw------T   1 root     root     5242880 Feb 27 23:43 test
# du -ak
5122    ./test
5124    .

Notice how du and ls agree on sizes.

9. Enable compresssion

zfs set compression=on usbdisk
# pwd
/usbdisk
# mkfile 5m testcompression
# ls -l
total 10246
-rw------T   1 root     root     5242880 Feb 27 23:43 test
-rw------T   1 root     root     5242880 Feb 27 23:48 testcompression
# du -ak
5122    ./test
0       ./testcompression
5124    .

 Notice that ls shows a 5 MB file but du -ak shows a zero size file because zero filled files compress so well.

10.  Now remove one of the USB memory sticks from the hub and attempt to create file.

# mkfile 5m test2
# zpool status

  pool: usbdisk
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        usbdisk       ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            c2t0d0p0  ONLINE       0     0     0
            c3t0d0p0  ONLINE       0   156     0
            c4t0d0p0  ONLINE       0     0     0

errors: No known data errors

zpool status reports that although a device is missing, data is intact.

Re-insert the removed memory stick and...

# zpool scrub usbdisk
# zpool status

  pool: usbdisk
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver completed with 0 errors on Thu Feb 28 00:37:03 2008
config:

        NAME          STATE     READ WRITE CKSUM
        usbdisk       ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            c2t0d0p0  ONLINE       0     0     0
            c3t0d0p0  ONLINE       0   254     0
            c4t0d0p0  ONLINE       0     0     0

errors: No known data errors
# zpool clear usbdisk
# zpool status

  pool: usbdisk
 state: ONLINE
 scrub: resilver completed with 0 errors on Thu Feb 28 00:37:03 2008
config:

        NAME          STATE     READ WRITE CKSUM
        usbdisk       ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            c2t0d0p0  ONLINE       0     0     0
            c3t0d0p0  ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0

errors: No known data errors

zpool scrub examines all data in the specified pools to verify that it checksums correctly. For  replicated  (mirror  or raidz)  devices,  ZFS  automatically  repairs any damage discovered during the scrub.

11.  Now for some real fun with export and import.

# cd /
# zpool export usbdisk
# zpool list

Note that the pool usbdisk is no longer listed.  Remove all three memory sticks.  Mix them up.  Re-insert them.

# zpool import
  pool: usbdisk
    id: 13155150575270542445
 state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        usbdisk       ONLINE
          raidz1      ONLINE
            c2t0d0p0  ONLINE
            c4t0d0p0  ONLINE
            c3t0d0p0  ONLINE
# zpool import usbdisk
# zpool status
 
  pool: usbdisk
 state: ONLINE
 scrub: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        usbdisk       ONLINE       0     0     0
          raidz1      ONLINE       0     0     0
            c2t0d0p0  ONLINE       0     0     0
            c4t0d0p0  ONLINE       0     0     0
            c3t0d0p0  ONLINE       0     0     0

errors: No known data errors

Notice how politely, ZFS tells you the name of the pool (even if you forgot it) and asks you to import it by name.  It doesn't matter that the actual "disks" have changed location.

12.  Transfer the disks to another systems (in this case a MacOS system). First note the files that exist and then export the file system. 

 On the Solaris system....

# ls -l
total 20473
-rw------T   1 root     root     5242880 Feb 28 00:32 test
-rw------T   1 root     root     5242880 Feb 28 00:49 testcompression
# du -a
10236   ./test
1       ./testcompression
20477   .
# cd /
# zpool export usbdisk

Shutdown the virtual machine and exit VMware to avoid confusion. Remove the USB hub from the Mac.

Now on Mac OS X 10.5 Re-insert the USB hub. MacOS X Finder produces an error: "Disk inserted was not readable by this computer."

Click "Ignore." Open the MacOS X terminal applications.

$ sudo -s
Password:
bash-3.2# zpool import
  pool: usbdisk
    id: 13155150575270542445
 state: ONLINE
status: The pool is formatted using an older on-disk version.
action: The pool can be imported using its name or numeric identifier, though
    some features will not be available without an explicit 'zpool upgrade'.
config:

    usbdisk     ONLINE
      raidz1    ONLINE
        disk4   ONLINE
        disk3   ONLINE
        disk5   ONLINE
bash-3.2# zpool import usbdisk
bash-3.2# cd /Volumes/usbdisk
bash-3.2# ls
test        testcompression
bash-3.2# du -a
10236    ./test
1    ./testcompression
10241    .
# zfs get all usbdisk
NAME     PROPERTY       VALUE                  SOURCE
usbdisk  type           filesystem             -
usbdisk  creation       Thu Feb 28  0:32 2008  -
usbdisk  used           5.14M                  -
usbdisk  available      200M                   -
usbdisk  referenced     5.03M                  -
usbdisk  compressratio  1.00x                  -
usbdisk  mounted        yes                    -
usbdisk  quota          none                   default
usbdisk  reservation    none                   default
usbdisk  recordsize     128K                   default
usbdisk  mountpoint     /Volumes/usbdisk       default
usbdisk  sharenfs       off                    default
usbdisk  checksum       on                     default
usbdisk  compression    on                     local
usbdisk  atime          on                     default
usbdisk  devices        on                     default
usbdisk  exec           on                     default
usbdisk  setuid         on                     default
usbdisk  readonly       off                    default
usbdisk  zoned          off                    default
usbdisk  snapdir        hidden                 default
usbdisk  aclmode        groupmask              default
usbdisk  aclinherit     secure                 default
usbdisk  canmount       on                     default
usbdisk  shareiscsi     off                    default
usbdisk  xattr          on                     default
usbdisk  copies         1                      default

Like magic, the USB-based ZFS array is now accessible (read-only) to MacOS X 10.5.  A future update is expected to support R/W access. The compression property is still turned on as it was in Solaris.

PS.  I tried mounting the devices in Solaris using Virtual Box by Innotek (recently acquired by Sun).  This software for MacOS X is currently in Beta test.  I received some rather nasty messages about: Failing to create proxy device for USB device.  Virtual Box also runs on Linux, Windows and OpenSolaris hosts.

 See here what Constantin has done with Virtual Box on Open Solaris with ZFS.

Using the ZFS GUI.

I used the command line but ZFS also has a fully capable browser interface.  To use it the webconsole service must be enabled:

# svcadm enable webconsole

Point your browser to:  https://localhost:6789.  Login with the root username and password.