Friday August 22, 2008

POFACS Podcast: Home Servers are quickly becoming Commonplace

I remember having talked at a conference 3 years ago and predicting that home servers are going to become a central part of most people's homes. Today, this would not be a surprise, but back then, running a server at home was really only for computer geeks.

Now, the entertainment industry gives us many home server alternatives to choose from: Add 50-100 EUR to a USB disk's price, and you'll get a built in server that offers the space to your local network through SMB, NFS or other protocols. Microsoft has discovered this, too and they're busily debugging their Windows Home Server products. UPnP has emerged as a standard for driving audio/video components over the network from servers, be they beefed up USB disks or some machine running some OS with some server component or a real dedicated home server machine. If you use iTunes and enable the "sharing" piece, you're already running a home server.

Of course, this is all driven by clients. First, people imported their music from CDs into their computers so they could listen on the go and fill their MP3 players. Then, they discovered that running a PC or even a laptop in your living room to listen to your music isn't really cool and lacks that WAF that makes or breaks most living room decisions. Soon, specialized living room clients started to pop up, such as the Roku Soundbridge or the Logitech SqueezeBox. Digital TV set-top-boxes and PVRs like the DreamBox were also early adopters of the home network by either offering TV streams on the network or using network attached storage for storing recorded TV shows. And the current generation of game consoles comes with Wifi and/or wired networking as a central part of their strategy, and they make good network media players as well. Even the traditional vendors of home entertainment equipment such as TVs, Hifi systems etc. have started to adopt some way of accepting digital audio and/or video from the network for A/V Receivers, DVD-Players, TVs etc. My current favourite, for example is the Linn Sneaky Music DS. And I applaud them for boldy migrating their records business to the digital world, in full studio master quality. You can even buy their full music catalog pre-installed on a 2TB NAS storage appliance, including UPnP server!

The current edition of the POFACS Podcast (sorry, it's in German) talks about the various ways a home server can add value to your living room experience, from serving files to your family's laptops, being a backup repository to the more interesting topics of serving music for dinner in a WAF-friendly way or handling your TV recordings over the net so you don't have to worry about noisy PCs and harddisks sitting in your living room. Enjoy!

"POFACS Podcast: Home Servers are quickly becoming Commonplace" has been brought to you by Constantin's Blooog.
This entry was created on 2008-08-22 04:14:36.0 PST and is associated with the following tags:
home it podcast podcasting pofacs server solaris
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.

Wednesday August 13, 2008

ZFS Replicator Script, New Edition

About a year ago, I blogged about a useful script that handles recursive replication of ZFS snapshots across pools. It helped me migrate my pool from a messy configuration into the clean two-mirrored-pairs configuration I have now.

Meanwhile, the fine guys at the ZFS developer team introduced recursive send/receive into the ZFS command, which makes most of what the script does a simple -F flag to the zfs(1M).

Unfortunately, this new version of the ZFS command has not (yet?) been ported back to Solaris 10, so my ZFS snapshot replication script is still useful for Solaris 10 users, such as Mike Hallock from the School of Chemical Sciences at the University of Illinois at Urbana-Champaign (UIUC). He wrote:

Your script came very close to exactly what I needed, so I took it upon myself to make changes, and thought in the spirit of it all, to share those changes with you.

The first change he in introduced was the ability to supply a pattern (via -p) that selects some of the potentially many snapshots that one wants to replicate. He's a user of Tim Foster's excellent automatic ZFS snapshot service like myself and wanted to base his migration solely on the daily snapshots, not any other ones.

Then, Mike wanted to migrate across two different hosts on a network, so he introduced the -r option that allows the user to specify a target host. This option simply pipes the replication data stream through ssh at the right places, making ZFS filesystem migration across any distance very easy.

The updated version including both of the new features is available as zfs-replicate_v0.7.tar.bz2. I didn't test this new version but the changes look very good to me. Still: Use at your own risk.

Thanks a lot, Mike! 

"ZFS Replicator Script, New Edition" has been brought to you by Constantin's Blooog.
This entry was created on 2008-08-13 13:25:49.0 PST and is associated with the following tags:
open opensolaris opensource remote replication script snapshot solaris source zfs zpool
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [3]

Tuesday August 12, 2008

ZFS saved my data. Right now.

As you know, I have a server at home that I use for storing all my photos, music, backups and more using the Solaris ZFS filesystem. You could say that I store my life on my server.

For storage, I use Western Digital's MyBook Essential Edition USB drives because they are the cheapest ones I could find from a well-known brand. The packaging says "Put your life on it!". How fitting.

Last week, I had a team meeting and a colleague introduced us to some performance tuning techiques. When we started playing with iostat(1M), I logged into my server to do some stress tests. That was when my server said something like this:

constant@condorito:~$ zpool status

(data from other pools omitted)

  pool: santiago
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: scrub completed after 16h28m with 0 errors on Fri Aug  8 11:19:37 2008
config:

	NAME         STATE     READ WRITE CKSUM
	santiago     DEGRADED     0     0     0
	  mirror     DEGRADED     0     0     0
	    c10t0d0  DEGRADED     0     0   135  too many errors
	    c9t0d0   DEGRADED     0     0    20  too many errors
	  mirror     ONLINE       0     0     0
	    c8t0d0   ONLINE       0     0     0
	    c7t0d0   ONLINE       0     0     0

errors: No known data errors

This tells us 3 important things:

• Two of my disks (c10t0d0 and c9t0d0) are happily giving me garbage back instead of my data. Without knowing it.
  Thanks to ZFS' checksumming, we can detect this, even though the drive thinks everything is ok.
  No other storage device, RAID array, NAS or file system I know of can do this. Not even the increasingly hyped (and admittedly cool-looking) Drobo [1].
• Because both drives are configured as a mirror, bad data from one device can be corrected by reading good data from the other device. This is the "applications are unaffected" and "no known data errors" part.
  Again, it's the checksums that enable ZFS to distinguish good data blocks from bad ones, and therefore enabling self-healing while the system is reading stuff from disk.
  As a result, even though both disks are not functioning properly, my data is still safe, because (luckily, albeit with millions of blocks per disk, statistics is on my side here) the erroneous blocks don't overlap in terms of what pieces of data they store.
  Again, no other storage technology can do this. RAID arrays only kick in when the disk drives as a whole are unacessible or when a drive  diagnoses itself to be broken. They do nothing against silent data corruption, which is what we see here and what all people on this planet that don't use ZFS (yet) can't see (yet). Until it's too late.
  
• Data hygiene is a good thing. Do a "zpool scrub <poolname>" once in a while. Use cron(1M) to do this, for example every other week for all pools.

Over the weekend, I ordered myself a new disk (sheesh, they dropped EUR 5 in price already after just 5 days...) and after a "zpool replace santiago c10t0d0 c11t0d0" on monday, my pool started resilvering:

constant@condorito:~$ zpool status

(data from other pools omitted)

  pool: santiago
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-9P
 scrub: resilver in progress for 1h13m, 6.23% done, 18h23m to go
config:

        NAME           STATE     READ WRITE CKSUM
        santiago       DEGRADED     0     0     0
          mirror       DEGRADED     0     0     0
            replacing  DEGRADED     0     0     0
              c10t0d0  DEGRADED     0     0   135  too many errors
              c11t0d0  ONLINE       0     0     0
            c9t0d0     DEGRADED     0     0    20  too many errors
          mirror       ONLINE       0     0     0
            c8t0d0     ONLINE       0     0     0
            c7t0d0     ONLINE       0     0     0

errors: No known data errors

The next step for me is to send the c10t0d0 drive back and ask for a replacement under warranty (it's only a couple of months old). After receiving c10's replacement, I'll consider sending in c9 for replacement (depending on how the next scrub goes).

Which makes me wonder: How will drive manufacturers react to a new wave of warranty cases based on drive errors that were not easily detectable before?

[1] To the guys at Drobo: Of course you're invited to implement ZFS into the next revision of your products. It's open source. In fact, Drobo and ZFS would make a perfect team!

"ZFS saved my data. Right now." has been brought to you by Constantin's Blooog.
This entry was created on 2008-08-12 06:44:22.0 PST and is associated with the following tags:
corruption data drobo integrity opensolaris solaris storage zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [5]