Tuesday May 27, 2008

OpenSolaris Home Server: ZFS and USB Disks

A couple of weeks ago, OpenSolaris 2008.05, project Indiana, saw its first official release. I've been looking forward to this moment so I can upgrade my home server and work laptop and start benefiting from the many cool features. If you're running a server at home, why not use the best server OS on the planet for it?

This is the first in a small series of articles about using OpenSolaris for home server use. I did a similar series some time ago and got a lot of good and encouraging feedback, so this is an update, or a remake, or home server 2.0, if you will.

I'm not much of a PC builder, but Simon has posted his experience with selecting hardware for his home server. I'm sure you'll find good tips there. In my case, I'm still using my trusty old Sun Java W1100z workstation, running in my basement. And for storing data, I like to use USB disks.

USB disk advantages

This is the moment where people start giving me that "Yeah, right" or "Are you serious?" looks. But USB disk storage has some cool advantages:

• It's cheap. About 90 Euros for half a TB of disk from a major brand. Can't complain about that.
• It's hot-pluggable. What happens if your server breaks and you want to access your data? With USB it's as easy as unplug from broken server, plug into laptop and you're back in business. And there's no need to shut down or open your server if you just want to add a new disk or change disk configuration.
• It scales. I have 7 disks running in my basement. All I needed to do to make them work with my server was to buy a cheap 15 EUR 4-port USB card to expand my existing 5 USB ports. I still have 3 PCI slots left, so I could add 12 disks more at full USB 2.0 speed if I wanted.
• It's fast enough. I measure about 10MB/s in write performance with a typical USB disk. That's about as fast as you can get over a 100 MBit/s LAN network which most people use at home. As long as the network remains the bottleneck, USB disk performance is not the problem.

ZFS and USB: A Great Team

But this is not enough. The beauty of USB disk storage lies in its combination with ZFS. When adding some ZFS magic to the above, you also get:

• Reliability. USB disks can be mirrored or used in a RAID-Z/Z2 configuration. Each disk may be unreliable (because they're cheap) individually, but thanks to ZFS' data integrity and self-healing properties, the data will be safe and FMA will issue a warning early enough so disks can be replaced before any real harm can happen.
• Flexibility. Thanks to pooled storage, there's no need to wonder what disks to use for what and how. Just build up a single pool with the disks you have, then assign filesystems to individual users, jobs, applications, etc. on an as-needed basis.
• Performance. Suppose you upgrade your home network to Gigabit Ethernet. No need to worry: The more disks you add to the pool, the better your performance will be. Even if the disks are cheap.

Together, USB disks and ZFS make a great team. Not enterprise class, but certainly an interesting option for a home server.

ZFS & USB Tips & Tricks

So here's a list of tips, tricks and hints you may want to consider when daring to use USB disks with OpenSolaris as a home server:

• Mirroring vs. RAID-Z/Z2: RAID-Z (or its more reliable cousin RAID-Z2) is tempting: You get more space for less money. In fact, my earlier versions of zpools at home were a combination of RAID-Z'ed leftover slices with the goal to squeeze as much space as possible at some reliability level out of my mixed disk collection.
  But say you have a 3+1 RAID-Z and want to add some more space. Would you buy 4 disks at once? Isn't that a bit big, granularity-wise?
  That's why I decided to keep it simple and just mirror. USB disks are cheap enough, no need to be even more cheap. My current zpool has a pair of 1 TB USB disks and a pair of 512 GB USB disks and works fine.
  Another advantage of this aproach is that you can organically modernize your pool: Wait until one of your disks starts showing some flakyness (FMA and ZFS will warn you as soon as the first broken data block has been repaired). Then replace the disk with a bigger one, then its mirror with the same, bigger size. That will give you more space without the complexity of too many disks and keep them young enough to not be a serious threat to your data. Use the replaced disks for scratch space or less important tasks.
• Instant replacement disk: A few weeks ago, one of my mirrored disks showed its first write error. It was a pair of 320GB disks, so I ordered a 512GB replacement (with the plan to order the second one later). But now, my mirror may be vulnerable: What if the second disk starts breaking before the replacement has arrived?
  That's why having a few old but functional disks around can be very valuable: In my case, took a 200GB and a 160GB disk and combined them into their own zpool:
  

  zpool create temppool c11t0d0 c12t0d0

  Then, I created a new ZVOL sitting on the new pool:
  

  zfs create -sV 320g temppool/tempvol

  Here's out temporary replacement disk! I then attached it to my vulnerable mirror:
  

  zfs attach santiago c10t0d0 /dev/zvol/dsk/temppool/tempvol

  And voilá, my precious production pool stated resilvering the new virtual disk. After the new disk arrived and has been resilvered, the temporary disk can be detached, destroyed and its space put to some other good use.
  Storage virtualization has never been so easy!
• Don't forget to scrub: Especially with cheap USB disks, regular scrubbing is important. Scrubbing will check each and every block of your data on disk and make sure it's still valid. If not, it will repair it (since we're mirroring or using RAID-Z/Z2) and tell you what disk had a broken block so you can decide whether it needs to be replaced or not just yet.
  How often you want to or should scrub depends on how much you trust your hardware and how much your data is being read out anyway (any data that is read out is automatically checked, so that particular portion of the data is already "scrubbed" if you will). I find scrubbing once every two weeks a useful cycle, othery may prefer once a month or once a week.
  But scrubbing is a process that needs to be initiated by the administrator. It doesn't happen by itself, so it is important that you think of issuing the "zpool scrub" command regularly, or better, set up a cronjob for it to happen automatically.
  As an example, the following line:
  

  23 01 1,15 * * for i in `zpool list -H -o name`; do zpool scrub $i; done

  in your crontab will start a scrub for each of your zpools twice a month on the 1st and the 15th at 01:23 AM.
• Snapshot often: Snapshots are cheap, but they can save the world if you accientally deleted that important file. Same rule as with scrubbing: Do it. Often enough. Automatically. Tim Foster did a great job of implementing an automatic ZFS snapshot service, so why don't you just install it now and set up a few snapshot schemes for your favourite ZFS filesystems?
  The home directories on my home server are snapshotted once a month (and all snapshots are kept), once a week (keeping 52 snapshots) and once a day (keeping 31 snapshots). This gives me a time-machine with daily, weekly and monthly granularities depending on how far back in time I want to travel through my snapshots.

So, USB disks aren't bad. In fact, thanks to ZFS, USB disks can be very useful building blocks for your own little cost-effective but reliable and integrity-checked data center.

Let me know what experiences you made while using USB storage at home, or with ZFS and what tips and tricks you have found to work well for you. Just enter a comment below or send me email!

"OpenSolaris Home Server: ZFS and USB Disks" has been brought to you by Constantin's Blooog.
This entry was created on 2008-05-27 13:40:54.0 PST and is associated with the following tags:
disks howto opensolaris server solaris usb zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [11]

Monday May 26, 2008

HELDENFunk Podcast featured in "Blick über den Tellerrand"

Our HELDENFunk podcast, part of the german Systemhelden.com sysadmin portal (If you don't understand German, you may prefer systemheroes.co.uk) has been featured in episode #166 of Alex Wunschel's "Blick über den Tellerrand". Watch out after minute 21:50.

The "Blick" is a weekly podcast in german about the "Blogosphere, Podosphere, Web x.0 and user/corporate-generated knick-knack". If you understand german and are interested in how social media is conquering Germany, this podcast is a must-listen. The german saying "Blick über den Tellerrand" means "glance across the edge of the plate", which is the german version of "glancing beyond one's nose".

The hot topic discussed in this and the preceding episodes is about Germany's public broadcasting agencies. On one hand, they get money from everybody who owns a radio, TV or a computer (read: Everyone, like a tax) and they're supposed to use it to create high-quality programming. On the other hand, the current draft of their "Rundfunkstaatsvertrag" (broadcasting state contract) forbids them to use more than 5% of the budget for online media. Their stance in this dilemma is published in the form of a controversial documentary called "Quoten, Klicks und Kohle" which can be loosely translated to "Vieweing Figures, Clicks and Dough".

You and I, but not enough people apparently, know that all media is significantly moving towards online ways of distribution. In fact, according to a study made by Bonn University and IBM, classic TV is losing importance, in particular among the younger generations and may become less siginificant than online media quite soon.

As part of this discussion, Alex is receiving quite a lot of feedback via email, phone and as MP3 files, which is where the HELDENFunk podcast is being mentioned in the current episode.

But who is this "Kontainer Kalle" guy?

"HELDENFunk Podcast featured in "Blick über den Tellerrand"" has been brought to you by Constantin's Blooog.
This entry was created on 2008-05-26 01:49:23.0 PST and is associated with the following tags:
germany heldenfunk podcast podcasting podpimp systemhelden systemhelden.com tellerrand
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.

Tuesday May 20, 2008

Geek Marketing

This morning, I listened to the "Blick über den Tellerrand" podcast, where Alex Wunschel gives us his thoughts and findings on the "Blogosphere, Podosphere, Web x.0 and User/Corporate-generated Knicknack". In one of his latest episodes, he interviewed Steve Rubel. 

Steve Rubel is a PR expert and author of one of the most read blogs called "Micropersuasion". In one of his articles, published through his employer Edelman Digital, he lists nine digital trends for the future. One of the trends that caught my attention is "Geek Marketing".

In a blog post, he further explains the concept: Technology is moving so fast, that marketing divisions are increasingly hiring geeks to help them understand developments in IT better. A geek marketer therefore is a link between technology and marketing people.

Cool. That's what I've been doing over the last year or so. I just didn't know there was a name for it! 

At Sun, we have a lot of geeks (they're mostly called "SEs" or "System Engineers") and we don't have much marketing (we'd rather spend the money on creating cool technology such as ZFS, UltraSPARC T2 or project Blackbox to name a few). But those few marketers we have, are really at the edge of the digital age. Starting from our long history of employee blogs, through our presence on Second Life to the Systemhelden.com portal (with the HELDENFunk podcast) - Our marketing people know what's hot and cool in the digital world, and how to engage the Sun geek communities to help them make cool stuff happen.

This fits very well with the book I'm currently reading: Dan Pink "A Whole New Mind". In this book, Pink argues that the virtues of the left half of the brain (typically associated with logical thinking) are not enough for today's global and networked world, in which left-brain work is increasingly outsourced to Asia and other emerging countries. In other words: If you want to keep your job, you better start thinking about your right half of your brain.

Right-brain work is quite interesting. Pink introduces six new aptitudes: Design, Story, Symphony, Empathy, Play, Meaning are senses that complement the Information Age worker's logical skills and help him be successful in the new "Conceptual Age".

So, without knowing, by participating in all this video, podcasting, blogging and Web 2.0 stuff, studying better ways of doing presentations and gaming on a Wii, DS or a Playstation, we're actually training the right half of our brains.

Who knew work could actually be that much fun?

Edit:

Added a link to the actual Tellerrand episode. Bummer!

P.S.: The photo shows me in front of a lot of technology. I find this quite fitting the geek theme. The technology is actually a nuclear fusion reactor in Garching. Yes, the kind of stuff you see in superhero movies right before someone gets a new superpower. Didn't seem to work on me, though. 

"Geek Marketing" has been brought to you by Constantin's Blooog.
This entry was created on 2008-05-20 14:10:28.0 PST and is associated with the following tags:
geek marketing podcasting web2.0
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [1]

Friday May 16, 2008

Detlef Drewanz on Virtualization in the POFACS Podcast

If you understand german, are interested in virtualization and listen to podcasts, don't miss the current episode of the POFACS podcast.

POFACS, the podcast for alternative computer systems is a german podcast that coveres everything non-mainstream in computing. From people running their business on a Commodore 64 to the state of the art Amiga OS to office packages that fit on a floppy disk or one of the many Linux variants.

There have been a few episodes covering Solaris related technologies, such as ZFS and Project Indiana. Today adds an interview with my colleague Detlef from Berlin about virtualization.

Actually, whenever I listen to one of the POFACS episodes about some crazy new operating system that's being developed somewhere, I've always liked to try it out and see how it is. The perfect way to do that of course is to use virtualization, so you don't have to re-install your machine again. Well, that's where Sun's VirtualBox comes in: It comes with a great range of supported operating systems so there's a good chance it will run even the strangest alternative OS just fine.

But now, let me download Detlef's interview myself and listen to is. Enjoy!

"Detlef Drewanz on Virtualization in the POFACS Podcast" has been brought to you by Constantin's Blooog.
This entry was created on 2008-05-16 07:52:28.0 PST and is associated with the following tags:
german podcast pofacs solaris virtualization
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.

Friday May 02, 2008

Favourite Free Fun Geek Cartoons to Cheer you up!

World economy bad? Financial results lower than expected? Stock price down the toilet? Or just bad weather?

No need to worry, last time I checked, after rain, always came the Sun, and it was stronger than ever!

Meanwhile, let me cheer you up with some favourite geek cartoons of mine:

User Friendly

This strip depicts the life of the heroic employees of an ISP called "Columbia Internet". It's something like the Dilbert of sysadmins, if you will. Lots of fun references to geek culture. And if you travel often by plane, you'll enjoy the strip above, it's ah, so true... For more background, read the Wikipedia entry on User Friendly.

XKCD

Above you see the future of the Solaris Network Auto-Magic (NWAM) feature. How could the author know? xkcd is "a webcomic of romance, sarcasm, math, and language", and very funny. It became famous for it's depiction of online communities as a world map. Read more about xkcd in its about page.

Geek and Poke

...and now you know the real reason why Peter and I like to drive Enterprise 2.0 at Sun :). Geek and Poke is another self-published cartoon by a guy from Hamburg called Oliver Widder.

All of these cartoons come with a license to make them redistributable, so I'm glad I could put some of my favourite strips on this page and I hope they have cheered you up a bit :). If you want some more fun, it's easy: Just click on the cartoons above to see more. But be careful, they are addictive...

What other great geek cartoons did I miss? What are your favourites? Leave a comment!

"Favourite Free Fun Geek Cartoons to Cheer you up!" has been brought to you by Constantin's Blooog.
This entry was created on 2008-05-02 13:42:54.0 PST and is associated with the following tags:
comics fun geek humour
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.