Friday Jul 04, 2008
Friday Jul 04, 2008
A Hands on Introduction to ZFS Pools; Part 2: Mirrors, Disk Failures and Spare Disks
The small hands on tutorial shows how to manage disks through ZFS with very limited resources. The tutorial is based on the following components
-
One 7 port USB 2.0 hub
-
Six 1GB USB 2.0 memory sticks
It has been released in multiple parts:
- Getting started, pool creation import and export of pools
- Mirrors, managing disk failures and spare disks
- RAIDZ2, resilvering and scrubbing
Within this tutorial you will learn
-
How to create a mirror in ZFS
-
How to expand it
-
How to replace broken components
As already stated in the first part; get some extra graphical tools like iobar from the freely available Tools CD. This will visualize nicely what's happening on your disks (see below):
Getting Started
Non Disclaimer: Please be cautious with your system. All
commands apply significant changes to your system. Data corruption may
easily occur. Have a good backup of your system. The system is not
warranted to work since it depends on the USB sticks, the hub and how
how Solaris get's informed about status changes. ZFS hasn't been
optimized for USB sticks and hick ups may occur at any time. Do not use
such a configuration for important work. This is a low cost self
learning sand box!
As already stated in the first part:
-
Become super user (su)
-
Clean up your /dev/dsk device list (devfsadm -C)
Plug in your first 2 memory sticks:
The rmformat reveils the following mappin:
-
stick 1: c4t0d0p0
-
stick 2: c5t0d0p0
Creating a mirrored pool
Create a mirrored pool with the pool name mpool by typing:
jack@opensolaris:~# zpool create mpool mirror c4t0d0p0 c5t0d0p0
Check the status of our mirror with:
jack@opensolaris:~# zpool status mpool
pool: mpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
errors: No known data errors
The mirror got mounted automaticallyand it is available through the directory /mpool.
Keep the file system under stress. Create a tar file from the /usr file tree in the background:
jack@opensolaris: cd /mpool
jack@opensolaris: tar -cf usr.tar /usr &
Enlarging a mirrored pool
The next step is to add two more disks to enhance the file system.
Plug in memory stick 3 and 4:
rmformat will reveil that the following mapping
-
stick 3: c6t0d0p0
-
stick 4: c7t0d0p0
Keep in mind. This will vary system by system.
Enlarge the mirror by typing:
jack@opensolaris:~# zpool add mpool mirror c6t0d0p0 c7t0d0p0
Validate the operation with zpool status:
jack@opensolaris:~# zpool status mpool
pool: mpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c6t0d0p0 ONLINE 0 0 0
c7t0d0p0 ONLINE 0 0 0
errors: No known data errors
Your system has now twice the capacity. The IO operation is still going on in the back ground. Use your last 2 memory sticks to configure spare disks for potential failures:.
Plug in stick 5 and 6:
The configuration is now complete. The picture documents as well that stick 4 is currently busy.
rmformat documents the following mapping
-
stick 5: c8t0d0p0
-
stick 6: c9t0d0p0
Add your two disks as spares with the following command:
jack@opensolaris:~# zpool add mpool spare c8t0d0p0 c9t0d0p0
pool: mpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c6t0d0p0 ONLINE 0 0 0
c7t0d0p0 ONLINE 0 0 0
spares
c8t0d0p0 AVAIL
c9t0d0p0 AVAIL
errors: No known data errors
Now we're able to start with the real fun. Disk pulling! Pull stick 3 as showed on the picture below:
Now check with zpool status on what's going on:
ack@opensolaris:~# zpool status -v mpool
pool: mpool
state: DEGRADED
scrub: resilver completed after 0h0m with 0 errors on Thu Jul 3
05:02:06 2008
config:
NAME STATE READ WRITE CKSUM
mpool DEGRADED 0 0 0
mirror DEGRADED 0 0 0
c4t0d0p0 REMOVED 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c6t0d0p0 ONLINE 0 0 0
c7t0d0p0 ONLINE 0 0 0
spares
c8t0d0p0 AVAIL
c9t0d0p0 AVAIL
errors: No known data errors
Controller 4 is recognized as gone and there is no data corruption. Everything is still operational but the redundancy is gone. We now have to replace the memory stick gone with a spare one. This happens through:
jack@opensolaris:~# zpool replace mpool c4t0d0p0 c8t0d0p0
The spare disks is now getting resilvered. The progress may be checked with the zpool status command. The case below documents a resilvering in progress:
jack@opensolaris:~# zpool status -v mpool
pool: mpool
state: DEGRADED
status: One or more devices is currently being resilvered. The
pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h0m, 11.57% done, 0h3m to go
config:
NAME STATE READ WRITE CKSUM
mpool DEGRADED 0 0 0
mirror DEGRADED 0 0 0
spare DEGRADED 0 0 0
c4t0d0p0 REMOVED 0 0 0
c8t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c6t0d0p0 ONLINE 0 0 0
c7t0d0p0 ONLINE 0 0 0
spares
c8t0d0p0 INUSE currently in use
c9t0d0p0 AVAIL
errors: No known data errors
The spare disk c8t0d0p0 is now in use. It's getting worse now. We pull out stick 4 as well:
c7t0d0p0 now needs to be replace with the remaining spare disk c9t0d0p0. zpool status shows the fix after the resilvering:
jack@opensolaris:~# zpool replace mpool c7t0d0p0 c9t0d0p0
pool: mpool
state: DEGRADED
scrub: resilver completed after 0h9m with 0 errors on Thu Jul 3
05:32:50 2008
config:
NAME STATE READ WRITE CKSUM
mpool DEGRADED 0 0 0
mirror DEGRADED 0 0 0
spare DEGRADED 0 0 0
c4t0d0p0 REMOVED 0 0 0
c8t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror DEGRADED 0 0 0
c6t0d0p0 ONLINE 0 0 0
spare DEGRADED 0 0 0
c7t0d0p0 REMOVED 0 0 0
c9t0d0p0 ONLINE 0 0 0
spares
c8t0d0p0 INUSE currently in use
c9t0d0p0 INUSE currently in use
errors: No known data errors
Plugging the remaining sticks back in lead to the following:
The remaining disks will be made online and resilvered:
jack@opensolaris:~# zpool status -v mpool
pool: mpool
state: ONLINE
scrub: resilver completed after 0h2m with 0 errors on Thu Jul 3
06:18:46 2008
config:
NAME STATE READ WRITE CKSUM
mpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c4t0d0p0 ONLINE 0 0 0
c5t0d0p0 ONLINE 0 0 0
mirror ONLINE 0 0 0
c6t0d0p0 ONLINE 0 0 0
c7t0d0p0 ONLINE 0 0 0
spares
c8t0d0p0 AVAIL
c9t0d0p0 AVAIL
errors: No known data errors
This concludes the second part of the ZFS hands an exercises.
You should have learned:
-
How to create a mirror
-
How to expand it
-
How to replace broken components
Released up to now are:
-
Mirrors, managing disk failures and spare disks
- RAIDZ2, resilvering and scrubbing
The is a typo in the tar command (/use vs /usr).
Posted by Wayne Abbott on July 04, 2008 at 05:32 PM CEST #
Thank you for the feed back!
The typo got corrected.
My mother told me not to lie...
This command was a command I entered in a separate window which wasn't part of my log under the script command.
Posted by Stefan Schneider on July 05, 2008 at 06:18 PM CEST #