Thursday December 03, 2009

OSDevCon 2009 Paper: Implementing a simple ZFS Auto-Scrub Service with SMF, RBAC, IPS and Visual Panels Integration - Lessons learned

A while ago, I wrote a little tool that helps you keep your ZFS pools clean by automatically running regular scrubs, similar to what the OpenSolaris auto-snapshot service does.

The lessons I learned during development of this service went into an OSDevCon 2009 paper that was presented in September 2009 in Dresden. It is a nice summary of things to keep in mind when developing SMF services of your own and it includes a tutorial on writing a GUI based on the OpenSolaris Visual Panels project.

Check out the Whitepaper here, the slides here, the SMF service here and if you want to take a peek at the Service's Visual Panels Java code, you'll find it here.

"OSDevCon 2009 Paper: Implementing a simple ZFS Auto-Scrub Service with SMF, RBAC, IPS and Visual Panels Integration - Lessons learned" has been brought to you by Constantin's Blooog.
This entry was created on 2009-12-03 03:15:28.0 PST and is associated with the following tags:
2009 ips opensolaris osdevcon panels paper rbac smf solaris visual whitepaper zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [0]

Monday November 16, 2009

Fun With DTrace: The Windows-Key Prank

The current episode of the German HELDENFunk podcast features an interview with Chris Gerhard about one of his favourite subjects: DTrace (in English, beginning at 14:58):

After the interview, we hear a guy called "Konteener Kalle" express his love (in German) for DTrace by playing a prank on his boss: Whenever he presses the Windows key (on an OpenSolaris system, mind you), he's punished by watching the XScreensaver BSOD hack (of course not knowing that it's just a screensaver).

That little joke challenged me to actually implement this prank. Here's how to do it.

The Idea

The idea of this prank is to start the XScreensaver Blue-Screen-of-Death screensaver (which simulates a Windows crash experience) on an OpenSolaris system whenever the user presses a certain key a certain number of times. This could be the Windows-Key (which doesn't have any real use on an OpenSolaris machine) or any other key. We count the number of key presses and only execute the BSOD after a certain number of key presses in order to make the prank less obvious.

Step 1: Identify the Windows (or any other) Key

If you have a Windows-Keyboard, this is easy: Run xev and press the Windows-Key. Take note of the keycode displayed in the xev output. Of course you can use any other key as well to play this prank. In this case, I'm using the left Control-Key, because I don't have a Windows-Key on the system I'm working on. The Control key has the keycode 37.

Step 2: Configure XScreensaver for BSOD

XScreensaver comes with a great collection of "hacks" that do interesting stuff on the screen when the screensaver activates. Check out the /usr/lib/xscreensaver/hacks directory. Each hack can be run individually, but then it will only execute inside a new window. For the BSOD illusion to be realistic, we want to execute the BSOD hack in full-screen.

This can be achieved by telling XScreensaver to demo the BSOD hack for us. It will then create a full-screen window and execute the BSOD hack inside the new window. The following command will tell XScreensaver to run a hack for us:

xscreensaver-command -demo <number>

The <number> part is a little complicated: XScreensaver looks at its config file ~/.xscreensaver where it stores a list of programs and arguments after the keyword "programs:". <number> simply refers to the number of the hack on that list. Therefore, we must create an entry in our admin user's .xscreensaver file that starts bsod(6) with the right parameters and that gives us a known number to call xscreensaver-command with.

Let's put our entry at the top of the list so we can simply use the number "1" to execute the BSOD screensaver. Somewhere in our .xscreensaver, the programs section should look like this:

  ...
  textFile:       /etc/motd
  textProgram:    date
  textURL:        http://blogs.sun.com/roller/rss

  programs:                                                                     \
  -               "BSOD Windoze"  bsod -root -only nt         \n\
  -                "Qix (solid)"  qix -root -solid -segments 100              \n\
  -          "Qix (transparent)"  qix -root -count 4 -solid -transparent      \n\
  ...

You can test this by running xscreensaver-command -demo 1.

Step 3: Write a DTrace Script That Sets Up the Trap

Now it gets more interesting. How do we use DTrace to find out when a user presses a certain key? All we know is that the Xorg server processes the keystrokes for us. So let's start by watching Xorg in action. The following DTrace command will trace all function calls within Xorg:

pfexec dtrace -n pid`pgrep Xorg`:::entry'{ @func[probefunc] = count(); }'

Let's start it, press the desired key 10 times, then stop it with CTRL-C. You'll see a long list of Xorg functions, sorted by the number of times they've been called. Since we pressed the key 10 times, it's a good idea to look for functions that have been called ca. 10 times. And here, we seem to be lucky:

  ...
  miUnionO                                                          8
  DeviceFocusInEvents                                               9
  CommonAncestor                                                   10
  ComputeFreezes                                                   10
  CoreLeaveNotifies                                                10
  key_is_down                                                      11
  FreeScratchPixmapHeader                                          12
  GetScratchPixmapHeader                                           12
  LookupIDByType                                                   12
  ProcShmDispatch                                                  12
  ProcShmPutImage                                                  12
  ...

The key_is_down function looks like exactly the function we're looking for! In fact, some googling tells us that this function's 2nd argument is the keycode of the key that is down when the function is called.

Why do we see "11" and not "10" function calls to key_is_down? Because it also counted my pressing of the Ctrl-Key when I stopped the DTrace script through Ctrl-C :).

This gives us enough knowledge to create the following DTrace script:

  #!/usr/sbin/dtrace -s

  /*
   * BSODKey.d
   */

  /*
   * This D script will monitor a certain key in the system. When this key is
   * pressed, a shell script will be executed that simulates a BSOD.
   *
   * The script needs the process id of the Xorg server to tap into as its
   * first argument.
   *
   * One example of using this script is to punish a user pressing the
   * Windows key on an OpenSolaris system by launching the BSOD screen saver.
   */

  #pragma D option quiet
  #pragma D option destructive

  BEGIN
  {
          ctrlcount = 0;
          keycode=37
  }

  pid$1::key_is_down:entry
  /arg1 == keycode/
  {
          ctrlcount ++;
  }

  pid$1::key_is_down:return
  /ctrlcount == 10/
  {
          ctrlcount = 0;
          system("/usr/bin/xscreensaver-command -demo 1");
  }

First, we need to enable DTrace's destructive mode (ever heard of a "constructive prank"?) otherwise we can't call the system-command at the end. The script uses the pid provider to tap into Xorg. Therefore, we need to give it the PID of the Xorg server as an argument:

pfexec ./BSODKey.d `pgrep Xorg`

It then sets up a probe that fires whenever key_is_down is called with our keycode and counts the key presses. At the end of the key_is_down function call, it checks whether we reached 10 keypresses, then executes the BSOD screen saver and resets the counter. You may need to make sure that the DISPLAY variable is set correctly for the BSOD program to show up on the victim's screen when starting this script.

After hitting the Control-Key 10 times, we're rewarded with our beloved BSOD:

Conclusion

That wasn't too difficult, was it? Yes, one could have done the same thing by writing a regular script that taps into /dev/kbd or something similar. But the beauty of DTrace lies in the simplicity of this script (Tap into the right function while it's running) and in the fact that it now can be modified very easily to fire BSODs at any kind of event, including the user hitting a certain area of the screen with his mouse or selecting a particular text or whatever you choose it to be.

So, have fun with this script and let me know in the comments what kind of pranks (or helpful actions) you can imagine with DTrace!

"Fun With DTrace: The Windows-Key Prank" has been brought to you by Constantin's Blooog.
This entry was created on 2009-11-16 08:32:14.0 PST and is associated with the following tags:
bsod dtrace fun heldenfunk scripting windows
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.

Sunday October 25, 2009

A Small and Energy-Efficient OpenSolaris Home Server

In an earlier entry, I outlined my most important requirements for an optimal OpenSolaris Home Server. It should:

1. Run OpenSolaris in order to fully leverage ZFS,
2. Support ECC memory, so data is protected at all times,
3. Be power-efficient, to help the environment and control costs,
4. Use a moderate amount of space and be quiet, for some extra WAF points.

So I went shopping and did some research on possible components. Here's what I came up with:

Choosing a Platform: AMD or Intel?

Disclosure: My wife works for AMD, so I may be slightly biased. But I think the following points are still very valid.

Intel is currently going through a significant change in architecture: The older Core 2 microarchitecture was based on the Front Side Bus (FSB), where the CPU connects to the Northbridge which contains the memory controller and connects to the memory, while also connecting to the Southbridge which connects to I/O.

Now, they are switching to the new Nehalem microarchitecture which has a memory-controller built into the CPU and a scalable I/O bus called Quickpath Interconnect that connects CPUs with other CPUs and/or IO.

Unfortunately, none of these architectures seem to support ECC memory at consumer level pricing. The cheapest Intel-based ECC-motherboard I could find had still more than double the cost of an AMD-based one. Even though the new Intel Core i7 series is based on Nehalem and thus could support ECC memory easily in theory, Intel somehow chose to not expose this feature. In addition, Core i7 CPUs are relatively new and there are not yet any power efficient versions available.

The Intel Atom processor series may be interesting for a home server from a pure power-saving perspective, but again, Atom motherboards don't support ECC and once your workload becomes a little more demanding (like transcoding or some heavier compiling), you'll miss the performance of a more powerful CPU.

AMD on the other hand has a number of attractive points for the home server builder:

• AMD consumer CPUs use the same microarchitecture than their professional CPUs (currently, it's the K10 design). They only vary by number of cores, cache size, number of HT channels, TDP and frequency, which are all results of the manufacturing process. All other microarchitecture features are the same. When using an AMD consumer CPU, you essentially get a "smaller brother" of their high end CPUs.
• This means you'll also get a built-in memory-controller that supports ECC.
• This also means less chips to build a system (no Northbridge needed) and thus lower power-consumption.
• AMD has been using the HyperTransport Interconnect for quite a while now. This is a fast, scaleable interconnect technology that has been on the market for quite a while so chipsets are widely available, proven and low-cost.

So it was no suprise that even low-cost AMD motherboards at EUR 60 or below are perfectly capable of supporting ECC memory which gives you an important server feature at economic cost.

My platform conclusion: Due to ECC support, low power consumption and good HyperTransport performance at low cost, AMD is an excellent platform for building a home server.

AMD Athlon II X2 240e: A Great Home Server CPU

While I was shopping around for AMD Athlon CPUs and just before I was about to decide for an AMD Athlon II X2 variant, AMD offered me one of their brand new AMD Athlon II X2 240e for testing, provided that I blogged about it. Thank you, AMD!

Introduced in October 20th, this CPU is part of the newest energy-efficient range of consumer CPUs from AMD. It has 2 cores (hence X2), snazzy 2.8 GHz and a 2 MB L2 cache. What's most important: The TDP for this CPU is only 45W, meaning that even under the highest stress, this CPU will never exceed 45W of power consumption. Including the memory controller. As you've guessed already, the "e" in the model number stands for "efficient".

There's an important trade-off to consider for home server CPUs: For instance, the AMD Phenom II series would have been more powerful because it has an additional L3 cache, but their TDP is at 65W minimum. While big caches (both with AMD and Intel) are good for compute-intensive operations and games, they can't help much in a home server context: Home servers spend most of their non-idle time transferring data from A to B (files, videos, music) and a cache doesn't help much here, cause it's just another stop between I/O and CPU to pass by. Transferred data hardly gets re-used.

Instead, for home servers, sacrificing the L3 cache for lower power consumption makes a lot of sense: You pay less for the CPU and you pay less for your power bill without sacrificing too much (if any) server relevant performance.

My CPU conclusion: For home servers, AMD Athlon II "e" series are perfect, because they save power and money and do the job very well. For games you might choose a more powerful Phenom II processor, which delivers better compute power at a slightly higher power bill.

Finding the Right Motherboard

After nailing the Platform and CPU question, I needed a motherboard. This can be a confusing process: For each CPU there are different chipsets, then there are different vendors offering motherboards based on these chipset, and then they offer different variants with different features. What should a good home server motherboard offer?

• OpenSolaris support: Most motherboards "just work" with OpenSolaris, especially if they've been available for some time and/or use some well-known chipsets. To remove any doubts, consult the OpenSolaris Hardware Compatibility List.
• ECC: Yes, AMD's ECC support is in the CPU, but just in case, we want to make sure that the motherboard exposes this feature to the user.
• Integrated graphics: We only need a graphics card for installation, BIOS-settings/updates or debugging. The graphics shouldn't consume much power and doesn't need to deliver a lot of performance. Therefore, integrated graphics is just fine for a home server, and it saves some precious slot space, too.
• Enough SATA ports: You need two ports for a mirrored root pool, another two for a mirrored data pool and then some more for hot-spare or if you want to spread your data pool across 4 or more disks. Many motherboards come with 6 SATA ports, which is ok.
• Stability: If you have the choice between different variants, try to go for the "business" or "quality" version. It will likely have slightly better components and be better tuned towards 24/7 operation than the "performance", "gaming" or "overclocker" type boards which try to squeeze out more performance at the expense of durability.

Here's a very useful email thread on the OpenSolaris ZFS-discuss mailing list about CPU and motherboard options, pros and cons and user experiences. In this discussion, F.Wessels recommended the M3A78 series from Asus so I went for the M3A78-CM motherboard, which is their "business class" variant, priced at around 60 Euros and it has 6 SATA and 12(!) USB ports.

My motherboard conclusion: The Asus M3A78-CM motherboard has everything I need for a home server at a very low cost, and it's proven to run OpenSolaris just fine.

The Case: Antec NSK-1380

I won't go into much details about the case. My goal was to find one that can support at least 4 disks while being as compact as possible. The Antec NSK-1380 was the smallest case I could find that supports 4 disks. It comes with a built-in power supply, an extra fan, some features to help with keeping noise down and it looked ok for a PC case.

Miscellaneous Tips&Tricks

While putting everything together, I ran into some smaller issues here and there. Here's what I came up with to solve them:

• CPU cooler: My CPU being a gift from AMD, it came without a cooler. I chose the Zalman CNPS 7000C AL-CU because it was the cheapest cooler that was less than 6cm high, which is a limit imposed by the case. Unfortunately, it collides with one of the 4 harddisk slots by a mere couple of millimeters, so I need to figure out how to best cut or bend the blades to make room for the disk without hurting the cooler too much. I'm not very familiar with PC CPU coolers, but I suspect that with 45W TDP one could even get away with a passive cooler. The Zalman came with a throttle circuit which I set to the minimum speed. This seems to be more than enough and it makes the system really silent, but I need some more thorough testing to confirm. Drop me a comment if you are familiar with passive cooling of 45W TDP CPUs.
• Boot disks: We need something to boot our home server from, so boot disks are unavoidable. I say "disks", because they should always be mirrored. But other than providing a way of booting, they tend to get in the way, especially when space is a precious resource in a small case. Some people therefore boot from CF cards or other small flash media. This can be a nice solution, especially combined with an IDE-to-CF adapter, but consumer level flash media is either very slow (a typical OS install can take many hours!) or very expensive. While looking for alternatives, I found a nice solution: The Scythe Slot Rafter fits into an unused PCI slot (taking up the breadth of two) and provides space for mounting four 2.5" disks at just EUR 5. These disks are cheap, good enough and I had an unused one lying around anyway, so that was a perfect solution for me.
• Extra NIC: The Asus M3A78-CM comes with a Realtek NIC and some people complained about driver issues with OpenSolaris. So I followed the advice on the aforementioned Email thread and bought an Intel NIC which is well supported, just in case.
• USB boot: I couldn't get the M3A78-CM to boot from USB at all. I tried a USB stick and a USB disk and different boot settings in the BIOS to no avail. I gave up and built in a DVD-ROM drive from my old server just to install OpenSolaris, then built it out again. Jan Brosowski has the same motherboard and he found out that you need to update to the latest BIOS revision and use the USB port that's right below the Ethernet port for USB boot to work. YMMV :).

The Result

And now for the most important part: How much power does the system consume? I did some testing with one boot disk and 4GB of ECC RAM and measured about 45W idle. While stressing CPU cores, RAM and the disk with multiple instances of sysbench, I could not get the system to consume more than 80W. All in all, I'm very pleased with the numbers, which are about half of what my old system used to consume. I didn't do any detailed performance tests yet, but I can say that the system feels very responsive and compile runs just rush along the screen. CPU temperature won't go beyond the low 50Cs on a hot day, despite using the lowest fan speed, so cooling seems to work well, too.

I just started full 24/7 operation of my new home server this weekend, so I hope I'll have some more long-term experience about performance and stability in a few months. Meanwhile, I'm in the middle of configuring the system, installing some services and implementing a new way of managing my home server. But that's probably the topic of another blog post...

Do you agree with the home server conclusions I reached in this post? Or would you suggest alternatives? Do you have experiences to share with the mentioned components? Or do you have suggestions and tips on how to get the most out of them? Let me know by posting a comment here!

Many thanks go to Michael Schmid of AMD for sending me the AMD Athlon II X2 240e CPU.

"A Small and Energy-Efficient OpenSolaris Home Server" has been brought to you by Constantin's Blooog.
This entry was created on 2009-10-25 15:22:05.0 PST and is associated with the following tags:
amd athlon build home opensolaris server x2 zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [14]

Friday October 02, 2009

Gonzalez Goes GeekAndPoke: My First Co-Authored Webcomic

A while ago, I highlighted a few of my favourite web comics. Little did I know then, that today I was going to be part of one. Here's the story:

Yesterday, @moellus complained yet again about his eternal nemesis, the NT admin, by saying something like: "Damn, can't unfollow the NT admin because he doesn't twitter - here's why - that's the sh*# he sends me!" (Original Tweet)

That reminded me of the famous insult by the late Douglas Adams from "The Hitchhiker's Guide to the Galaxy", in which Arthur Dent says about the Vogons: "I wish I had a daughter so I could forbid her to marry one..."

So I twittered back that "I wish you were on Twitter so I could unfollow you!" must be the new worst insult you can do in 2009 and asked Oliver whether that would make a nice Geek And Poke cartoon.

Today, I'm proud to be part of the "Geek And Poke"-uversum, here's the cartoon:

I guess, to be "GeekAndPoked" is the new "Slashdotted" :). Thanks, Oliver!

P.S.: @moellus: Actually, to a geek, in 2009, sending someone a box of Windows 7 is like the medieval slapping with a glove. Don't take it lightly and get your LART-whip ready!

"Gonzalez Goes GeekAndPoke: My First Co-Authored Webcomic" has been brought to you by Constantin's Blooog.
This entry was created on 2009-10-02 00:55:40.0 PST and is associated with the following tags:
comic funny geek twitter webcomic
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [2]

Saturday September 19, 2009

New OpenSolaris ZFS Home Server: Requirements

A few months ago, I decided it was time for a new home server. The old one (see picture) is now more than 3 years old (the hardware is 2 years older), so it was time to plan ahead for the inevitable hardware failure. Curiously enough, my old server started to refuse working with some of my external USB disks only a few weeks ago, which confirmed my need for a new system. This is the beginning of a series of blog articles around building a new OpenSolaris home server.

Home Server Goals

Let's go over some goals for a home server to help us decide on the hardware. IMHO, a good home server should:

1. Run OpenSolaris. This means I don't want an appliance because this is too limiting and I'd end up hacking it to make it run something it doesn't do by itself anyway. It should therfore use a real OS.
   It also means I don't want to use Linux, because quite frankly the whole Linux landscape is too unstable, confused and de-focused (don't get me wrong: It's nice for experimentation and as a hobbyist-OS, but I want something more serious to guard my data).
   Windows is out of the question because it delivers too little for too high a price.
   I like BSD (and used to run NetBSD on my Amiga 4000 back then in the mid nineties), but it seems to be more oriented to some (albeit interesting) niches for my taste.
   Right now I prefer OpenSolaris because it's rock-solid, clean, well-documented, well-designed and it has lots of advanced features that other OSes only dream of. Yes, I'd still write and do the same if I weren't a Sun employee.
2. Leverage ZFS. This should be a no-brainer, but I just wanted to point out that any system that is serious about its data should absolutely run ZFS because of the end-to-end-integrity. Period. And then there are many useful features such as compression, send/receive, snapshots, ease of administration, no fscks and much more. Oh, and I'm looking forward to leveraging encryption and de-duplication at home in the near future, too!
3. Use ECC Memory: What's the use of having end-to-end data integrity with ZFS if your data is corrupted before ZFS can create it's checksum? That's why you need ECC Memory. Simply put: Use ECC memory and kiss those unexpected, unexplicable system crashes and broken data surprises good bye.
4. Be Power Efficient: Think 1.5 Euros of electricity bill per Watt per Year for a system running 24/7. The difference between your typical gaming PC and a power-efficient home server can easily be 50W or more when idle, so you're looking at an extra 75 Euros and more of free cash if you just pick your components more carefully. Notice that I'm not saying "Low-Power". There are a lot of compromises when trying to reach absolute low-powerness. Like many optimization problems, squeezing the last few Watts out of your system means investing a lot of money and effort while sacrificing important features. So I want this system to be power-efficient, but without too many sacrifices.
5. Use a Moderate Amount of Space: While my home server sits in the basement, form doesn't matter. But I may move into a new apartment where networking to the basement is not an option. Then the server needs to be living-room capable and have a decent WAF. Which also brings us to:
6. Be quiet: A power-efficient server needs less cooling which helps with being quiet. Again, we don't want to stretch the limits of quietness at all costs, but we want to make sure we don't do any obvious mistakes here that sacrifice the living-room capabilities of the system

What's Next

In the next blog entry, we'll discuss a few processor and platform considerations and reveal a cool, yet powerful option that presented itself to me. Meanwhile, feel free to check out other home server resources, such as Simon Breden's blog, Matthias Pfuetzner's blog, Jan Brosowski's Blog (German) or one of the many home server discussions on the zfs-discuss mailing list.

What are your requirements for a good home server? What do you currently use at home to fulfill your home server needs? What would you add to the above list of home server requirements? Feel free to add a comment below!

"New OpenSolaris ZFS Home Server: Requirements" has been brought to you by Constantin's Blooog.
This entry was created on 2009-09-19 08:24:29.0 PST and is associated with the following tags:
home opensolaris server solaris zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [6]

Thursday September 17, 2009

New OpenSolaris ZFS Auto-Scrub Service Helps You Keep Proper Pool Hygiene

One of the most important features of ZFS is the ability to detect data corruption through the use of end-to-end checksums. In redundant ZFS pools (pools that are either mirrored or use a variant of RAID-Z), this can be used to fix broken data blocks by using the redundancy of the pool to reconstruct the data. This is often called self-healing.

This mechanism works whenever ZFS accesses any data, because it will always verify the checksum after reading a block of data. Unfortunately, this does not work if you don't regularly look at your data: Bit rot happens and with every broken block that is not checked (and therefore not corrected), the probability increases that even the redundant copy will be affected by bit rot too, resulting in data corruption.

Therefore, zpool(1M) provides the useful scrub sub-command which will systematically go through each data block on the pool and verify its checksum. On redundant pools, it will automatically fix any broken blocks and make sure your data is healthy and clean.

It should now be clear that every system should regularly scrub their pools to take full advantage of the ZFS self-healing feature. But you know how it is: You set up your server and often those little things get overlooked and that cron(1M) job you wanted to set up for regular pool scrubbing fell off your radar etc.

Introducing the ZFS Auto-Scrub SMF Service

Here's a service that is easy to install and configure that will make sure all of your pools will be scrubbed at least once a month. Advanced users can set up individualized schedules per pool with different scrubbing periods. It is implemented as an SMF service which means it can be easily managed using svcadm(1M) and customized using svccfg(1M).

The service borrows heavily from Tim Foster's ZFS Auto-Snapshot Service. This is not just coding laziness, it also helps minimize bugs in common tasks (such as setting up periodic cron jobs) and provides better consistency across multiple similar services. Plus: Why invent the wheel twice?

Requirements

The ZFS Auto-Scrub service assumes it is running on OpenSolaris. It should run on any recent distribution of OpenSolaris without problems.

More specifically, it uses the -d switch of the GNU variant of date(1) to parse human-readable date values. Make sure that /usr/gnu/bin/date is available (which is the default in OpenSolaris).

Right now, this service does not work on Solaris 10 out of the box (unless you install GNU date in /usr/gnu/bin). A future version of this script will work around this issue to make it easily usable on Solaris 10 systems as well.

Download and Installation

You can download Version 0.5b of the ZFS Auto-Scrub Service here. The included README file explains everything you need to know to make it work:

After unpacking the archive, start the install script as a privileged user:

pfexec ./install.sh

The script will copy three SMF method scripts into /lib/svc/method, import three SMF manifests and start a service that creates a new Solaris role for managing the service's privileges while it is running. It also installs the OpenSolaris Visual Panels package and adds a simple GUI to manage this service.

After installation, you need to activate the service. This can be done easily with:

svcadm enable auto-scrub:monthly

or by running the GUI with:

vp zfs-auto-scrub

This will activate a pre-defined instance of the service that makes sure each of your pools is scrubbed at least once a month.

This is all you need to do to make sure all your pools are regularly scrubbed.

If your pools haven't been scrubbed before or if the time or their last scrub is unknown, the script will proceed and start scrubbing. Keep in mind that scrubbing consumes a significant amount of system resources, so if you feel that a currently running scrub slows your system too much, you can interrupt it by saying:

pfexec zpool scrub -s <pool name>

In this case, don't worry, you can always start a manual scrub at a more suitable time or wait until the service kicks in by itself during the next scheduled scrubbing period.

Should you want to get rid of this service, use:

pfexec ./install.sh -d

The script will then disable any instances of the service, remove the manifests from the SMF repository, delete the scripts from /lib/svc/method, remove the special role and the authorizations the service created and finally remove the GUI. Notice that it will not remove the OpenSolaris Visual Panels package in case you want to use it for other purposes. Should you want to get rid of this as well, you can do so by saying:

pkg uninstall OSOLvpanels

Advanced Use

You can create your own instances of this service for individual pools at specified intervals. Here's an example:

  constant@fridolin:~$ svccfg
  svc:> select auto-scrub
  svc:/system/filesystem/zfs/auto-scrub> add mypool-weekly
  svc:/system/filesystem/zfs/auto-scrub> select mypool-weekly
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> addpg zfs application
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/pool-name=mypool
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/interval=days 
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/period=7
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/offset=0
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> setprop zfs/verbose=false
  svc:/system/filesystem/zfs/auto-scrub:mypool-weekly> end
  constant@fridolin:~$ svcadm enable auto-scrub:mypool-weekly

This example will create and activate a service instance that makes sure the pool "mypool" is scrubbed once a week.

Check out the zfs-auto-scrub.xml file to learn more about how these properties work.

Implementation Details

Here are some interesting aspects of this service that I came across while writing it:

• The service comes with its own Solaris role zfsscrub under which the script runs. The role has just the authorizations and profiles necessary to carry out its job, following the Solaris Role-Based Access Control philosophy. It comes with its own SMF service that takes care of creating the role if necessary, then disables itself. This makes a future deployment of this service with pkg(1) easier, which does not allow any scripts to be started during installation, but does allow activation of newly installed SMF services.
• While zpool(1M) status can show you the last time a pool has been scrubbed, this information is not stored persistently. Every time you reboot or export/import the pool, ZFS loses track of when the last scrub of this pool occurred. This has been filed as CR 6878281. Until that has been resolved, we need to take care of remembering the time of last scrub ourselves. This is done by introducing another SMF service that periodically checks the scrub status, then records the completion date/time of the scrub in a custom ZFS property called org.opensolaris.auto-scrub:lastscrub in the pool's root filesystem when finished. We call this service whenever a scrub is started and it deactivates itself once it's job is done.
• As mentioned above, the GUI is based on the OpenSolaris Visual Panels project. Many thanks to the people on its discussion list to help me get going. More about creating a visual panels GUI in a future blog entry.

Lessons learned

It's funny how a very simple task like "Write an SMF service that takes care of regular zpool scrubbing" can develop into a moderately complex thing. It grew into three different services instead of one, each with their own scripts and SMF manifests. It required an extra RBAC role to make it more secure. I ran into some zpool(1M) limitations which I now feel are worthy of RFEs and working around them made the whole thing slightly more complex. Add an install and de-install script and some minor quirks like using GNU date(1) instead of the regular one to have a reliable parser for human-readable date strings, not to mention a GUI and you cover quite a lot of ground even with a service as seemingly simple as this.

But this is what made this project interesting to me: I learned a lot about RBAC and SMF (of course), some new scripting hacks from the existing ZFS Auto-Snapshot service, found a few minor bugs (in the ZFS Auto-Snapshot service) and RFEs, programmed some Java including the use of the NetBeans GUI builder and had some fun with scripting, finding solutions and making sure stuff is more or less cleanly implemented.

I'd like to encourage everyone to write their own SMF services for whatever tools they install or write for themselves. It helps you think your stuff through, make it easy to install and manage, and you get a better feel of how Solaris and its subsystems work. And you can have some fun too. The easiest way to get started is by looking at what others have done. You'll find a lot of SMF scripts in /lib/svc/method and you can extract the manifests of already installed services using svccfg export. Find an SMF service that is similar to the one you want to implement, check out how it works and start adapting it to your needs until your own service is alive and kicking.

If you happen to be in Dresden for OSDevCon 2009, check out my session on "Implementing a simple SMF Service: Lessons learned" where I'll share more of the details behind implementing this service including the Visual Panels part.

Edit (Sep. 21st) Changed the link to CR 6878281 to the externally visible OpenSolaris bug database version, added a link to the session details on OSDevCon.

"New OpenSolaris ZFS Auto-Scrub Service Helps You Keep Proper Pool Hygiene" has been brought to you by Constantin's Blooog.
This entry was created on 2009-09-17 07:25:34.0 PST and is associated with the following tags:
opensolaris script scrub service smf solaris tool useful zfs zpool
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [4]

Monday July 06, 2009

How to Fix OpenSolaris Keyboard Irregularities with Virtual Box

Virtual Box is great: It allows you to install OS A on OS B for impressively large sets of A and B OSes and their permutations. Almost everything works smoothly and seamlessly between host and guest: Cut&Paste, File sharing, networking, USB pass-through, even seamless windows are supported.

But there's one little glitch that is still a little annoying, but apparently not annoying enough for someone else to have blogged about this before: Keyboard remapping on Mac OS X hosts.

The Problem

Simple problem: Macs are different than PCs (phew), but they have slightly different keyboard mappings (oops). Most notably, on my German keyboard, the "<" key at the bottom left on the Mac will yield "^" on OpenSolaris and vice versa. Same thing goes for "@", which is Right-Alt-L on the Mac, but Right-Alt-Q on PCs. Similar difficulties are encountered if you try to create a "|" pipe symbol or angular/curved brackets ("[]" and "{}" respectively).

Pressing the Right Keys

Usually no big deal. Close your eyes and blindly type what you would type on a PC and that'll give you a good hint at where the right keystrokes are. That works because Virtual Box actually maps the physical locations of the keys between host and guest, but not what's painted on them. So, with a little practice, you should be fine. But what happens if you can't quite remember what that PC keyboard looked like?

Last Friday I had an hour or so left and the playfulness of the problem got the better of me, so I decided to see if this can be fixed the Unix way. It's actually quite easy.

Searching for a cure

There are some helpful hints on the net, most notably Petr Hruska's entry on "Switching Keyboard Layout in Solaris", but it only deals with internationalization issues. What if you have the keyboard nationalities right, but individual keys are still different as in the Mac/PC case? Here's a step-by-step guide to help you with any keyboard remapping problem, plus a bonus table for OpenSolaris on Macbook users to get you started:

Xmodmap to the rescue

1. We're going to use xmodmap(1) to remap the keys on our keyboard. Check out the man-page to familiarize yourself with how it works.
2. See the keystrokes as OpenSolaris sees them: Use xev(6) to find out what keycodes belong to the keys you want to correct.
3. Check out what OpenSolaris is thinking about your problematic keys, either by testing them in a terminal or by checking your version of the standard Sun USB keyboard layouts.
4. Before you start modifying the current keyboard mapping, get the currently active one by saying something like:
   xmakemap > ~/.xmodmaprc.current
   Caution: There seems to be a bug in xmakemap that corrupts some of the entires. So, please use this only for reference but do not feed this file back into xmodmap (see later) or you'll likely make your keyboard unusable (until this bug is resolved).
5. Start editing your own remapping script for xmodmap:
   vi ~/.xmodmaprc
6. For each key you want to remap, copy it's keycode entry from the xmakemap output into your own remapping table and modify to taste. Be careful, some entries from xmakemap are broken, but you should be able to figure those out. Here's my current .xmodmaprc file as a reference:
   

   !
   ! Set up keys for a MacBook Pro running OpenSolaris on VirtualBox
   !
   !       Key   Unshifted       Shifted         AltGraph        AltGraph-Shifted
   !       ---   --------------- --------------- --------------- ----------------
   
   keycode  49 = less            greater
   keycode  94 = asciicircum     degree          asciicircum     degree
   keycode  14 = 5               percent         bracketleft
   keycode  15 = 6               ampersand       bracketright
   keycode  16 = 7               slash           bar             backslash
   keycode  17 = 8               parenleft       braceleft
   keycode  18 = 9               parenright      braceright
   keycode  24 = q               Q               q               Q
   keycode  46 = l               L               at
   keycode  57 = n               N               asciitilde
   

   This works well on my MacBook Pro, your mileage may vary.
7. You can activate your remapping by saying something like:
   xmodmap ~/.xmodmaprc
8. In case something goes wrong and you render your keyboard useless, you can restart your X server by pressing Ctrl-Alt-Backspace twice.
9. If you're happy with your remapping, you can automatically activate it on every login by using the System->Preferences->Sessions panel and adding an entry for the above xmodmap command there.

Conclusion

I hope this little exercise in some lesser known X-Windows commands (Hi Jörg) was useful for you, now you shouldn't need to worry too much about keyboard mapping inconsistencies any more.

If you want to learn a little more about modifying your keyboard, check out this section of the OpenSolaris docs.

The example keymap modifications above work well for me, but I'm sure I've forgotten a key or two. What other keys did you remap and why? Feel free to leave me a comment below.

"How to Fix OpenSolaris Keyboard Irregularities with Virtual Box" has been brought to you by Constantin's Blooog.
This entry was created on 2009-07-06 14:24:32.0 PST and is associated with the following tags:
keyboard opensolaris solaris virtualbox xmodmap
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.

Monday June 29, 2009

Online-Workshop: Besserer Klang mit wenig Aufwand von der niche09

This post is in German because it's about a Podcasting workshop in German language. If you want this workshop to be in English, feel free to gather a bunch of people and invite me to do it for you.

Am 20.6.2009 fand in München das Podcamp München statt, besser bekannt als niche09. An diesem Samstag trafen sich über 100 Podcast-Begeisterte in München und tauschten sich zu verschiedenen Themen rund um's Podcasting aus. Das Programm bot einen schönen Querschnitt durch das Thema und im http://www.niche09.de/">niche09-Blog kann man sich die Workshops noch in Form von verschiedenen Aufzeichnungen auch nachträglich und online kostenlos zu Gemüte führen. An dieser Stelle vielen Dank an Alex Wunschel, die Sponsoren und die vielen Helfer, die diese wirklich schöne Konferenz zustande gebracht haben!

Alex war auch so nett, mich einen Workshop zum Thema "Besserer Klang mit wenig Aufwand: Tipps & Tricks beim Podcast-Produzieren" moderieren zu lassen. Ein Audio-Mitschnitt samt synchroner Folien ist nun als Video erhältlich, in der Hoffnung, dass dieser Workshop auch online vielen Leuten bei der Produktion ihrer Podcasts helfen möge:

Den Workshop könnt Ihr unten direkt anschauen, als Quicktime-Video für den Rechner oder als iPhone-Video herunterladen, sowie Euch die Folien zum Workshop anschauen.

Hier noch ein paar Links, Anmerkungen und Korrekturen zum Workshop. Keine Angst, ich bekomme von keinem der genannten Hersteller irgendwas, sondern spreche nur aus eigener Erfahrung bzw. verlässlichen Quellen.

• Nicht wundern, der "halbstündige Workshop" ist nur ein Witz, weil die Konferenz mit ca. 30 Min. Verspätung angefangen hat. Der Workshop war von vornherein auf 1 Stunde angelegt :).
• Für mobile Aufnahmen ist das Zoom H2 und sein größerer Bruder Zoom H4 von Samson sehr beliebt. Für vergleichsweise wenig Geld erhält man eine sehr gute Aufnahme-Qualität und eine praktische, mobile Handhabung. Darüber hinaus kann das Gerät kann auch als gutes USB-Mikrofon dienen.Im Workshop lobte jemand auch den Audio-Recorder von Olympus (nicht sicher, ob dieses Modell gemeint war).
• Die USB-Audio-Interfaces von M-Audio sind gut und günstig und für den Einstieg sehr empfehlenswert. Nach einiger Zeit bin ich jedoch aufgrund eines Tests im Professional Audio-Magazin zum Native Instruments Audio Kontrol 1 gewechselt, das mich durch sehr gute, rauschfreie Audio-Qualität sowohl bei der Aufnahme als auch bei der Ausgabe über Kopfhörer und Aktivboxen beeindruckt hat.
• Tim Pritlove vom Chaos Radio Express und MobileMacs empfahl uns die Beyerdynamic DT 297 Headsets für die stressfreie Aufnahme von mehreren Podcastern auf einmal, da die Mikros guten Klang bieten, man jede Stimme einzeln aufnehmen kann und die Kopfhörer präzises Feedback für die Sprecher erlauben. Alleine das richtige Audio-Interface/Mischpult/Vorschaltgerät, das jedem einzelnen seinen eigenen Feedback-Kanal gönnt und gleichzeitig eine getrennte Aufnahme ermöglicht, scheint noch ein ungelöstes Problem zu sein. Vielleicht hilft ein eigener Mehrkanal-Kopfhörerverstärker?
• Im MacCast 2009.04.14 gibt es ein schönes Interview mit Heroes-Star David H. Lawrence XVII, der u.a. auch ein eigenes Studio betreibt und vom Radio kommend zum Podcaster geworden ist. Er hat viele nützliche Tipps parat und empfiehlt u.a. das Audio-Technica AT2020, insbesondere die USB-Variante AT2020 USB. Im Workshop hatte ich leider "Audio-Technica" mit "Behringer" als Hersteller verwechselt, ich bitte um Entschuldigung für die Verwirrung...
• Auch in unserem HELDENFunk-Podcast verwenden wir das Audio-Technica AT2020, sowie ein paar Røde NT5 und können diese sehr empfehlen. Mehr Details gibt es in einem eigenen HELDENFunk behind the Scenes-Artikel. Inzwischen haben wir unser Setup um ein Mark of the Unicorn (MOTU) 8pre 8-Fach Firewire Audio-Interface erweitert, das wir ebenfalls sehr empfehlen können.

Ich hoffe, dieser Workshop ist trotz der Länge von 1 Stunde für Euch nützlich. Schickt mir Euer Feedback, Fragen und Anregungen, bei der nächsten Konferenz (niche10?) bin ich gerne wieder dabei!

"Online-Workshop: Besserer Klang mit wenig Aufwand von der niche09" has been brought to you by Constantin's Blooog.
This entry was created on 2009-06-29 14:24:44.0 PST and is associated with the following tags:
audio german klang niche09 podcasting presentation produktion qualität ton
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [5]

Tuesday June 16, 2009

Paris in the Clouds: A CloudCamp Paris report

Last week, Eric Bezille invited me to Paris for a couple of Cloud Computing related meetings and to help out with CloudCamp Paris. Paris in the clouds, what a nice experience!

This was also a great opportunity to try out the audio recording features of my LiveScribe Pulse pen. This pen not only can record what you write (on special dot paper), it can also record what has been said while you write, creating links between the words you write and the points in time of the audio recording. Very cool. You can then tap on the words in your notebook and the pen will play back the associated audio. Great for conferences, and I wish I had had this pen during my university times :). You can also export your notes including the audio as a flash movie and share them on the net, which is what I'm going to do below.

Intro Session and Lightning Talks

The CloudCamp was kicked off by a representative of Institut Telecom, the location sponsor of CloudCamp Paris. Sam Johnston gave a short and sweet introduction to Clouds, providing some definitions, examples and also some contrarian views, finishing with a short video on how easy it is to set up your account in the cloud.

A series of lightning talks by the sponsors gave us some interesting news, insights and context for the conference:

• Eric Bezille from Sun showed us what's behind Sun's cloud activities.
• Arvid Fossen from Aserver.com talked about how they provide datacenters as a service to their clients. Wanna have your own cloud? Go buy it as a turnkey solution!
• Matthew Hugo (Not sure if I got that name right...) from Runmyprocess.com showed some nice examples of integration between different cloud services.
• Josh Fraser, VP of Business Development at Rightscale showed some impressive examples of how the cloud can neatly adjust to your business demand curve.
• Peter Martin from Orange Business Services showed us some pictures of his kids who use clouds based services today (Facebook anyone?), pointing out that when they'll grow up to be CEOs, CIOs and decision makers, they're most likely not going to operate their own datacenters. Food for thought for the sceptics who think Cloud Computing is just a temporary hype or not ready (yet) for prime time: Just wait 'til your kids grow up. It may happen sooner than that, though, given the enthusiasm of the more than 100 people in the room...
• Finally, Owen Garrett from Zeus provided a really good reason for using a software load balancer: Take back control of your application!

Here are two pencasts with audio and notes taken during the above lightning talks. The first one covers the intro until and including the Rightscale talk, the second one starts with the Orange talk and finishes with the Zeus talk.

The Unpanel

Cloud Architecture Session

Wrapping It All Up

After the breakouts, a surprisingly large number of attendees were still there despite being late into the evening to gather and listen to the summaries of the different sessions. Here's the recording, including some notes to help you navigate.

All in all, this was a great event. A big thank you to Eric and his team in Paris and the sponsors for setting this up! More than ever, it became clear to me how significant the trend towards cloud computing is and how many talented people are part of this community, driving the future of IT into the sky.

Update: Eric now published his own summary with a lot of background information. It's a great read, so check it out!

"Paris in the Clouds: A CloudCamp Paris report" has been brought to you by Constantin's Blooog.
This entry was created on 2009-06-16 13:57:08.0 PST and is associated with the following tags:
cloud community computing livescribe mysql notes paris pulse sun virtualization
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [4]

Monday June 15, 2009

OpenSolaris meets Mac OS X in Munich

Last Wednesday, Wolfgang and I had the honor to present at "Mac Treff München", Munich's local Mac User Group. There are quite a few touching points between OpenSolaris and Mac OS X, such as ZFS, DTrace and VirtualBox, we thought it would be a good idea to contact them out of our Munich OpenSolaris User Group and talk a little bit about OpenSolaris.

Breaking the Ice

We were a little bit nervous about what would happen. Do Mac people care about the innards of a different, seemingls non-GUIsh OS? Are they just fanboys or are they open to other people's technologies? Will talking about redundancy, BFU, probes and virtualization bore them to death?

Fortunately, the 30-40 people that attended the event proved to be a very nice, open and tolerant group. They let us talk about OpenSolaris in General including some of the nitty-grittyness of the development process, before we started talking about the features that are more interesting to Mac users. We then talked about ZFS, DTrace and VirtualBox:

ZFS for Mac OS X (or not (yet)?)

Explaining the principles behind ZFS to people who are only used to draging'n'dropping icons, shooting photos or video and using computers to get work done, without having to care about what happens inside, is not easy. We concentrated on getting the basics of the tree structure, copy-on-write, check-summing and using redundancy to self-heal while using real world examples and metaphors to illustrate the principles. Here's the deal: If you have lots of important data (photos, recording, videos, anyone?) and care about it (content creators...), then you need to be concerned about data availability and integrity. ZFS solves that, it's that simple. A little animation in the slides were quite helpful in explaining that, too :).

The bad news is that ZFS seems to have vanished from all of Apple's communication about the upcoming Mac OS X Snow Leopard release. That's really bad, because many developers and end-users were looking forward to take advantage of it.

The good news is that there are still ways to take advantage of ZFS as a Mac User: Run an OpenSolaris file server for archiving your data or using it as a TimeMachine store, or even run a small OpenSolaris ZFS Server inside your Mac through VirtualBox.

DTrace: A Mac Developer/Admin's Heaven, Albeit in Jails

Next, we dove a little bit into DTrace and how it makes the OS really transparent for admins, developers and users. In addition to the dtrace(1) command, Apple created a nice GUI called "Instruments" as part of their XCode development environment that leverages the DTrace infrastructure to collect useful data about your application in realtime.

Alas, as with ZFS, there's another downer, and this time it's more subtle: While you can enjoy the power of DTrace in Mac OS X now, it's still kinda crippled, as Adam Leventhal pointed out: Processes can escape the eyes of DTrace at will, which counters the absolute observability idea of DTrace quite massively. Yes, there are valid reasons for both sides of the debate, but IMHO, legal things should be enforced using legal means, and software should be treated as software, meaning it is not a reliable way of enforcing any license contracts - with or without powerful tools such as DTrace.

OpenSolaris for all: VirtualBox

Finally, a free present to the Mac OS X community: VirtualBox. I still get emails asking me to spend 80+ dollars on some virtualization software for my Mac. There are at least two choices in that price range: VMware Workstation and Parallels. Well, the good news is that you can save your 80 bucks and use VirtualBox instead.

This may not be new to you, since as a reader of my blog you've likely heard of VirtualBox before, but it's always amazing for me to see how slowly these things spread. So, after reading this article, do your Mac friends a favour and tell them they can save precious money buy just downloading VirtualBox instead of spending money on other virtualization solutions for the Mac. It's really that simple.

Indeed, this was the part where the attendees took most of their notes, and asked a lot of questions about (ZFS being a close first in terms of discussion/questions).

Conclusion

After our presentations, a lot of users came up and asked questions about how to install OpenSolaris on their hardware and on VirtualBox. Some even asked where to buy professional services for installing them an OpenSolaris ZFS fileserver in their company. The capabilities of ZFS clearly struck some chords inside the Mac OS X community, which is no wonder: If you have lots of Audio/Video/Photo data and care about quality and availability, then there's no way around FS.

I used this event as an excuse to try out keynote, which worked quite well for me, especially because it helped me create some easy to understand animations about the mechanics of ZFS. I also liked the automatic guides a lot which help you position elements on your slides very easily and seem to guess very well what your layout intentions were. I'd love the OpenOffice folks to check out Keynote's guides and see if they can come up with something similar. So, here's a Keynote version of my "OpenSolaris for Mac Users" slides as well as a PDF version (both in German) for you to check out and re-use if you like.

Update: Wolfgang's introductory slides are now available for download as well and Klaus, the organizer of the event, posted a review in the Mac Treff München Blog with some pictures, too.

"OpenSolaris meets Mac OS X in Munich" has been brought to you by Constantin's Blooog.
This entry was created on 2009-06-15 02:32:51.0 PST and is associated with the following tags:
community dtrace mac mucosug munich opensolaris os virtualbox x zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [7]