Sunday October 12, 2008
All
|
General
Sunday October 12, 2008 BarCamp Munich 2008 - Enterprise 2.0, Open Source and the Future of Technology
I'm astonished to see that I haven't blogged for so long. Sorry to my readers, it's been some very busy times lately, and I hope I can write more in the coming weeks. I also owe an apology to the people that pointed out a bug with my ZFS replicator script and cron(1M), I'll look into it and make it my next post.
Barcamp at Sun in Munich
Yesterday, I attended Barcamp Munich 2008 which was sponsored by Sun (among other cool sponsors) and so it took place in the Sun Munich offices.
I was surprised to see that both sessions I proposed were accepted, plus one about Open Source Software at Sun that my colleague Stefan proposed with some support by me.
You can find a list of sessions for Saturday and Sunday on the web and it pays off to check back regularly, as the wiki is filling up with more and more collateral information around each track.
So, here's a roundup of session descriptions, slides and other links and materials for those of you who attended my sessions or could not attend, in chronological order.
Enterprise 2.0 - From Co-Workers to Co-Creators
This session was similar to the talk I did at Webkongress Erlangen a few months ago.
We had about 20 people in the room and quite a fruitful discussion on how to motivate employees to use new tools, how to guide employee behaviour and the challenges of opening up a company and making it more transparent.
Feel free to glance through my Enterprise 2.0 slides or read an earlier blog entry on a related subject. Also, check out Peter Reiser's blog, he has a number of great articles from behind the scenes of our SunSpace collaboration project.
Open Source Software at Sun
Stefan Schneider proposed a session about great software products that are available from Sun for free, as open source. We went through his list from least well-known to most popular.
Obviously, MySQL, StarOffice and OpenSolaris were at the end, but the more interesting software products were those that made the attendees go "Oh, I didn't know that!". One example of this category was Lightning, a rich calendar client.
Stefan recently posted his slides into the Sun Startups blog, thanks, Stefan!
The Future of Technology in 10, 20, 30 Years and More
This was a spontaneous talk that I offered after having seen the Barcamp Munich wishlist where people asked for a session on future technology developments, their effects on society and how one can cope with it.
I took some slides from a couple of earlier talks I did a while ago on similar topics and updated it for the occasion. The updated "Future of Technology" slidedeck is in German, but if enough people are interested, I can provide a translated version as well.
We started by looking at Moore's Law as an indicator of technology development. In "The Age of Spiritual Machines", Ray Kurzweil, a well-known futurist, pointed out that this law also holds for technology prior to integrated circuits, all the way down to Charles Babbage's difference engine of the 19th century.
With that in mind, we can confidently extend Moore's Law into the future, knowing that even if traditional chip technology ceases to deliver on Moore's Law, other technologies will pick up and help us achieve even higher amounts of computing power per amount of money/space/energy. Again, Kurzweil points out that if we compare the amount of computational power that one can purchase for $1000 for a given year with the complexity of all neurons of a brain and their connections to neighbouring neurons at their typical firing frequency, then the 2020s will be an interesting decade.
Key technologies of the future will be: Genetics and Biotechnology, Robotics and Nanotechnology.
We watched a fascinating video about Molecular Visualizations of DNA (here's a longer, more complete version) that made us witnesses of DNA being replicated, right before our eyes, at a molecular level. It's amazing to see how mechanical this process looks, almost like industrial robots grinding away on ribbons of DNA, cutting pieces, replicating them, then splicing them back in. In the near future, we will see personalized medicine, based on our own DNA, and optimized for our individual needs as well as novel applications of biotechnology for clean energy, new materials and the assembly of early molecular machines.
Robotics are another fascinating area of technology and we're seeing more and more robots enter our day to day life. Industrial and military robots may be an "old hat", but did you know that today, millions of households are already using robots to vacuum their floory, mow their lawns or perform other routine work? And we will see many more robots in the future, I'm sure. Meanwhile, I'm happy to say that my Roomba robot indeed saves a lot of precious time while fulfilling my natural geeky desire for cool gadgetry.
Finally, Nanotechnology will open up a new category of advanced technology that will affect all aspects of human life, the environment and the world. We watched a vision of a future nanofactory that fits onto a common desk and is capable of manufacturing an advanced laptop with 100 hours of battery life and a billion CPUs. But nanotechnology can do much more: Highly efficient solar cells, clean water, lightweight spacecrafts, nanobots that clean up your bloodstream, more advanced versions of your organs, brain implants and extensions, virtual reality that is indistinguishable from real reality and much more.
Check out the Foresight Institute's introduction to nanotechnology for more information about this fascinating topic, including a free PDF download of K. Eric Drexler's book "Engines of Creation". Real engineers will probably want to take a look at his textbook "Nanosystems Molecular Machinery Manufacturing and Computation"
One controversial topic when discussing the future is the Technological Singularity. This is the point in time, where artificial intelligence becomes powerful enough to create new technology on its own, thereby accelerating the advancement of technology without human intervention. A discussion of this topic can be found in Kurzweil's newest book "The Singularity is Near".
Another great way to think about the future is to read Stefan Pernar's sci-fi thriller "Jame5 - A Tale of Good and Evil". This book starts in the best Michael Crichton style and then becomes a deep and thoughtful discussion around the philosophy of the future, when mankind confronts the powers of strong AI. You can buy the book or just download the PDF for free. Highly recommended.
One of my favourite citations is said to be an old chinese curse: "May you live in interesting times."
Many thanks to all the people that I met during, or attended my sessions at, Barcamp Munich 2008, it was a most interesting event.
Edit (Oct., 13th): Meanwhile, a few blog reactions are rolling in: Dirk wrote a nice summary on the Enterprise 2.0 session (in German) while Ralph summarized the Future technology session (German as well). I found them through Markus' Barcamp Munich 2008 session meta entry. Thanks to all! Also, Stefan has posted his slides from the open source talk, see above.
Edit (Oct. 14th): Here are some more notes from Stefan Freimark (in German). Thank you!
"BarCamp Munich 2008 - Enterprise 2.0, Open Source and the Future of Technology" has been brought to you by Constantin's Blooog.
This entry was created on 2008-10-12 14:14:27.0 PST and is associated with the following tags:
#bcmuc08
barcamp
bcmuc08
enterprise
enterprise2.0
future
futurism
law
moore
nanotechnology
open
opensolaris
opensource
singularity
source
space
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Comments [1]
Wednesday August 13, 2008 ZFS Replicator Script, New Edition
About a year ago, I blogged about a useful script that handles recursive replication of ZFS snapshots across pools. It helped me migrate my pool from a messy configuration into the clean two-mirrored-pairs configuration I have now.
Meanwhile, the fine guys at the ZFS developer team introduced recursive send/receive into the ZFS command, which makes most of what the script does a simple -F flag to the zfs(1M).
Unfortunately, this new version of the ZFS command has not (yet?) been ported back to Solaris 10, so my ZFS snapshot replication script is still useful for Solaris 10 users, such as Mike Hallock from the School of Chemical Sciences at the University of Illinois at Urbana-Champaign (UIUC). He wrote:
Your script came very close to exactly what I needed, so I took it upon myself to make changes, and thought in the spirit of it all, to share those changes with you.
The first change he in introduced was the ability to supply a pattern (via -p) that selects some of the potentially many snapshots that one wants to replicate. He's a user of Tim Foster's excellent automatic ZFS snapshot service like myself and wanted to base his migration solely on the daily snapshots, not any other ones.
Then, Mike wanted to migrate across two different hosts on a network, so he introduced the -r option that allows the user to specify a target host. This option simply pipes the replication data stream through ssh at the right places, making ZFS filesystem migration across any distance very easy.
The updated version including both of the new features is available as zfs-replicate_v0.7.tar.bz2. I didn't test this new version but the changes look very good to me. Still: Use at your own risk.
Thanks a lot, Mike!
"ZFS Replicator Script, New Edition" has been brought to you by Constantin's Blooog.
This entry was created on 2008-08-13 13:25:49.0 PST and is associated with the following tags:
open
opensolaris
opensource
remote
replication
script
snapshot
solaris
source
zfs
zpool
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Comments [3]
Tuesday February 19, 2008 VirtualBox and ZFS: The Perfect Team
I've never installed Windows in my whole life. My computer history includes systems like the Dragon 32, the Commodore 128, then the Amiga, Apple PowerBook (68k and PPC) etc. plus the occasional Sun system at work. Even the laptop my company provided me with only runs Solaris Nevada, nothing else. Today, this has changed.
A while ago, Sun announced the acquisition of Innotek, the makers of the open-source virtualization software VirtualBox. After having played a bit with it for a while, I'm convinced that this is one of the coolest innovations I've seen in a long time. And I'm proud to see that this is another innovative german company that joins the Sun family, Welcome Innotek!
Here's why this is so cool.
After having upgraded my laptop to Nevada build 82, I had VirtualBox up and running in a matter of minutes. OpenSolaris Developer Preview 2 (Project Indiana) runs fine on VirtualBox, so does any recent Linux (I tried Ubuntu). But Windows just makes for a much cooler VirtualBox demo, so I did it:
After 36 years of Windows freedom, I ended up installing it on my laptop, albeit on top of VirtualBox. Safer XP if you will. To the top, you see my VirtualBox running Windows XP in all its Tele-Tubby-ish glory.
As you can see, this is a plain vanilla install, I just took the liberty of installing a virus scanner on top. Well, you never know...
So far, so good. Now let's do something others can't. First of all, this virtual machine uses a .vdi disk image to provide hard disk space to Windows XP. On my system, the disk image sits on top of a ZFS filesystem:
# zfs list -r poolchen/export/vm/winxp
NAME USED AVAIL REFER MOUNTPOINT
poolchen/export/vm/winxp 1.22G 37.0G 20K /export/vm/winxp
poolchen/export/vm/winxp/winxp0 1.22G 37.0G 1.05G /export/vm/winxp/winxp0
poolchen/export/vm/winxp/winxp0@200802190836_WinXPInstalled 173M - 909M -
poolchen/export/vm/winxp/winxp0@200802192038_VirusFree 0 - 1.05G -
Cool thing #1: You can do snapshots. In fact I have two snapshots here. The first is from this morning, right after the Windows XP installer went through, the second has been created just now, after installing the virus scanner. Yes, there has been some time between the two snapshots, with lots of testing, day job and the occasional rollback. But hey, that's why snapshots exist in the first place.
Cool thing #2: This is a compressed filesystem:
# zfs get all poolchen/export/vm/winxp/winxp0
NAME PROPERTY VALUE SOURCE
poolchen/export/vm/winxp/winxp0 type filesystem -
poolchen/export/vm/winxp/winxp0 creation Mon Feb 18 21:31 2008 -
poolchen/export/vm/winxp/winxp0 used 1.22G -
poolchen/export/vm/winxp/winxp0 available 37.0G -
poolchen/export/vm/winxp/winxp0 referenced 1.05G -
poolchen/export/vm/winxp/winxp0 compressratio 1.53x -
...
poolchen/export/vm/winxp/winxp0 compression on inherited from poolchen
ZFS has already saved me more than half a gigabyte of precious storage capacity already!
Next, we'll try out Cool thing #3: Clones. Let's clone the virus free snapshot and try to create a second instance of Win XP from it:
# zfs clone poolchen/export/vm/winxp/winxp0@200802192038_VirusFree poolchen/export/vm/winxp/winxp1
# ls -al /export/vm/winxp
total 12
drwxr-xr-x 5 constant staff 4 Feb 19 20:42 .
drwxr-xr-x 6 constant staff 5 Feb 19 08:44 ..
drwxr-xr-x 3 constant staff 3 Feb 19 18:47 winxp0
drwxr-xr-x 3 constant staff 3 Feb 19 18:47 winxp1
dr-xr-xr-x 3 root root 3 Feb 19 08:39 .zfs
# mv /export/vm/winxp/winxp1/WindowsXP_0.vdi /export/vm/winxp/winxp1/WindowsXP_1.vdi
The clone has inherited the mountpoint from the upper level ZFS filesystem (the winxp one) and so we have everything set up for VirtualBox to create a second Win XP instance from. I just renamed the new container file for clarity. But hey, what's this?
Damn! VirtualBox didn't fall for my sneaky little clone trick. Hmm, where is this UUID stored in the first place?
# od -A d -x WindowsXP_1.vdi | more
0000000 3c3c 203c 6e69 6f6e 6574 206b 6956 7472
0000016 6175 426c 786f 4420 7369 206b 6d49 6761
0000032 2065 3e3e 0a3e 0000 0000 0000 0000 0000
0000048 0000 0000 0000 0000 0000 0000 0000 0000
0000064 107f beda 0001 0001 0190 0000 0001 0000
0000080 0000 0000 0000 0000 0000 0000 0000 0000
*
0000336 0000 0000 0200 0000 f200 0000 0000 0000
0000352 0000 0000 0000 0000 0200 0000 0000 0000
0000368 0000 c000 0003 0000 0000 0010 0000 0000
0000384 3c00 0000 0628 0000 06c5 fa07 0248 4eb6
0000400 b2d3 5c84 0e3a 8d1c 8225 aae4 76b5 44f5
0000416 aa8f 6796 283f db93 0000 0000 0000 0000
0000432 0000 0000 0000 0000 0000 0000 0000 0000
0000448 0000 0000 0000 0000 0400 0000 00ff 0000
0000464 003f 0000 0200 0000 0000 0000 0000 0000
0000480 0000 0000 0000 0000 0000 0000 0000 0000
*
0000512 0000 0000 ffff ffff ffff ffff ffff ffff
0000528 ffff ffff ffff ffff ffff ffff ffff ffff
*
0012544 0001 0000 0002 0000 0003 0000 0004 0000
Ahh, it seems to be stored at byte 392, with varying degrees of byte and word-swapping. Some further research reveals that you better leave the first part of the UUID alone (I spare you the details...), instead, the last 6 bytes: 845c3a0e1c8d, sitting at byte 402-407 look like a great candidate for an arbitrary serial number. Let's try changing them (This is a hack for demo purposes only. Don't do this in production, please):
# dd if=/dev/random of=WindowsXP_1.vdi bs=1 count=6 seek=402 conv=notrunc
6+0 records in
6+0 records out
# od -A d -x WindowsXP_1.vdi | more
0000000 3c3c 203c 6e69 6f6e 6574 206b 6956 7472
0000016 6175 426c 786f 4420 7369 206b 6d49 6761
0000032 2065 3e3e 0a3e 0000 0000 0000 0000 0000
0000048 0000 0000 0000 0000 0000 0000 0000 0000
0000064 107f beda 0001 0001 0190 0000 0001 0000
0000080 0000 0000 0000 0000 0000 0000 0000 0000
*
0000336 0000 0000 0200 0000 f200 0000 0000 0000
0000352 0000 0000 0000 0000 0200 0000 0000 0000
0000368 0000 c000 0003 0000 0000 0010 0000 0000
0000384 3c00 0000 0628 0000 06c5 fa07 0248 4eb6
0000400 b2d3 2666 6fbb c1ca 8225 aae4 76b5 44f5
0000416 aa8f 6796 283f db93 0000 0000 0000 0000
0000432 0000 0000 0000 0000 0000 0000 0000 0000
0000448 0000 0000 0000 0000 0400 0000 00ff 0000
0000464 003f 0000 0200 0000 0000 0000 0000 0000
0000480 0000 0000 0000 0000 0000 0000 0000 0000
*
0000512 0000 0000 ffff ffff ffff ffff ffff ffff
0000528 ffff ffff ffff ffff ffff ffff ffff ffff
*
0012544 0001 0000 0002 0000 0003 0000 0004 0000
Who needs a hex editor if you have good old friends od and dd on board? The trick is in the "conv=notruc" part. It tells dd to leave the rest of the file as is and not truncate it after doing it's patching job. Let's see if it works:
Heureka, it works! Notice that the second instance is running with the freshly patched harddisk image as shown in the window above.
Windows XP booted without any problem from the ZFS-cloned disk image. There was just the occasional popup message from Windows saying that it found a new harddisk (well observed, buddy!).
Thanks to ZFS clones we can now create new virtual machine clones in just seconds without having to wait a long time for disk images to be copied. Great stuff. Now let's do what everybody should be doing to Windows once a virus scanner is installed: Install Firefox:
I must say that the performance of VirtualBox is stunning. It sure feels like the real thing, you just need to make sure to have enough memory in your real computer to support both OSes at once, otherwise you'll run into swapping hell...
BTW: You can also use ZFS volumes (called ZVOLs) to provide storage space to virtual machines. You can snapshot and clone them just like regular file systems, plus you can export them as iSCSI devices, giving you the flexibility of a SAN for all your virtualized storage needs. The reason I chose files over ZVOLs was just so I can swap pre-installed disk images with colleagues. On second thought, you can dump/restore ZVOL snapshots with zfs send/receive just as easily...
Anyway, let's see how we're doing storage-wise:
# zfs list -rt filesystem poolchen/export/vm/winxp
NAME USED AVAIL REFER MOUNTPOINT
poolchen/export/vm/winxp 1.36G 36.9G 21K /export/vm/winxp
poolchen/export/vm/winxp/winxp0 1.22G 36.9G 1.05G /export/vm/winxp/winxp0
poolchen/export/vm/winxp/winxp1 138M 36.9G 1.06G /export/vm/winxp/winxp1
Watch the "USED" column for the winxp1 clone. That's right: Our second instance of Windows XP only cost us a meager 138 MB on top of the first instance's 1.22 GB! Both filesystems (and their .vdi containers with Windows XP installed) represent roughly a Gigabyte of storage each (the REFER column), but the actual physical space our clone consumes is just 138MB.
Cool thing #4: ZFS clones save even more space, big time!
How does this work? Well, when ZFS creates a snapshot, it only creates a new reference to the existing on-disk tree-like block structure, indicating where the entry point for the snapshot is. If the live filesystem changes, only the changed blocks need to be written to disk, the unchanged ones remain the same and are used for both the live filesystem and the snapshot.
A clone is a snapshot that has been marked writable. Again, only the changed (or new) blocks consume additional disk space (in this case Firefox and some WinXP temporary data), everything that is unchanged (in this case nearly all of the WinXP installation) is shared between the clone and the original filesystem. This is de-duplication done right: Don't create redundant data in the first place!
That was only one example of the tremenduous benefits Solaris can bring to the virtualization game. Imagine the power of ZFS, FMA, DTrace, Crossbow and whatnot for providing the best infrastructure possible to your virtualized guest operating systems, be they Windows, Linux, or Solaris. It works in the SPARC world (through LDOMs), and in the x86/x64 world through xVM server (based on the work of the Xen community) and now joined by VirtualBox. Oh, and it's free and open source, too.
So with all that: Happy virtualizing, everyone. Especially to everybody near Stuttgart.
"VirtualBox and ZFS: The Perfect Team" has been brought to you by Constantin's Blooog.
This entry was created on 2008-02-19 13:18:18.0 PST and is associated with the following tags:
cool
hack
howto
innotek
open
opensolaris
opensource
solaris
virtualbox
virtualization
windows
zfs
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Comments [9]
Tuesday November 27, 2007 Shrink big presentations with ooshrink
I work in an environment where people use presentations a lot. Of course, we like to use StarOffice, which is based on OpenOffice for all of our office needs.
Presentation files can be big. Very big. Never-send-through-email-big. Especially, when they come from marketing departments and contain lots of pretty pictures. I just tried to send a Sun Systems overview presentation (which I created myself, so less marketing fluff), and it still was over 22MB big!
So here comes the beauty of Open Source, and in this case: Open Formats. It turns out, that OpenOffice and StarOffice documents are actually ZIP files that contain XML for the actual documents, plus all the image files that are associated with it in a simple directory structure. A few years ago I wrote a script that takes an OpenOffice document, unzips it, looks at all the images in the document's structure and optimizes their compression algorithm, size and other settings based on some simple rules. That script was very popular with my colleagues, it got lost for a while and thanks to Andreas it was found again. Still, colleagues are asking me about "That script, you know, that used to shrink those StarOffice presentations." once in a while.
Today, I brushed it up a little, teached it to accept the newer od[ptdc] extensions and it still works remarkably well. Here are some examples:
- The Sun homepage has a small demo presentation with a few vacation photos. Let's see what happens:
bash-3.00$ ls -al Presentation_Example.odp
-rw-r--r-- 1 constant sun 392382 Mar 10 2006 Presentation_Example.odp
bash-3.00$ ooshrink -s Presentation_Example.odp
bash-3.00$ ls -al Presentation_Example.*
-rw-r--r-- 1 constant sun 337383 Nov 27 11:36 Presentation_Example.new.odp
-rw-r--r-- 1 constant sun 392382 Mar 10 2006 Presentation_Example.odpWell, that was a 15% reduction in file size. Not earth-shattering, but we're getting there. BTW: The -s flag is for "silence", we're just after results (for now).
On BigAdmin, I found a presentation with some M-Series config diagrams:
bash-3.00$ ls -al Mseries.odp
-rw-r--r-- 1 constant sun 1323337 Aug 23 17:23 Mseries.odp
bash-3.00$ ooshrink -s Mseries.odp
bash-3.00$ ls -al Mseries.*
-rw-r--r-- 1 constant sun 379549 Nov 27 11:39 Mseries.new.odp
-rw-r--r-- 1 constant sun 1323337 Aug 23 17:23 Mseries.odpNow we're getting somewhere: This is a reduction by 71%!
- Now for a real-world example. My next victim is a presentation by Teera about JRuby. I just used Google to search for "site:sun.com presentation odp", so Teera is completely innocent. This time, let's take a look behind the scenes with the -v flag (verbose):
bash-3.00$ ooshrink -v jruby_ruby112_presentation.odp
Required tools "convert, identify" found.
ooshrink 1.2
Check out "ooshrink -h" for help information, warnings and disclaimers.
Creating working directory jruby_ruby112_presentation.36316.work...
Unpacking jruby_ruby112_presentation.odp...
Optimizing Pictures/1000020100000307000000665F60F829.png.
- This is a 775 pixels wide and 102 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 947, New: 39919. We better keep the original.
Optimizing Pictures/100000000000005500000055DD878D9F.jpg.
- This is a 85 pixels wide and 85 pixels high JPEG file.
- We will try re-encoding this image with JPEG quality setting of 75%.
- Failure: Old: 2054, New: 2089. We better keep the original.
Optimizing Pictures/1000020100000419000003C07084C0EF.png.
- This is a 1049 pixels wide and 960 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 99671, New: 539114. We better keep the original.
Optimizing Pictures/10000201000001A00000025EFBC8CCCC.png.
- This is a 416 pixels wide and 606 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 286677, New: 349860. We better keep the original.
Optimizing Pictures/10000000000000FB000001A6E936A60F.jpg.
- This is a 251 pixels wide and 422 pixels high JPEG file.
- We will try re-encoding this image with JPEG quality setting of 75%.
- Success: Old: 52200, New: 46599 (-11%). We'll use the new picture.
Optimizing Pictures/100000000000055500000044C171E62B.gif.
- This is a 1365 pixels wide and 68 pixels high GIF file.
- This image is too large, we'll resize it to 1280x1024.
- We will convert this image to PNG, which is probably more efficient.
- Failure: Old: 2199, New: 39219. We better keep the original.
Optimizing Pictures/100000000000019A000002D273F8C990.png.
- This is a 410 pixels wide and 722 pixels high PNG file.
- This picture has 50343 colors, so JPEG is a better choice.
- Success: Old: 276207, New: 32428 (-89%). We'll use the new picture.
Patching content.xml with new image file name.
Patching styles.xml with new image file name.
Patching manifest.xml with new image file name.
Optimizing Pictures/1000000000000094000000E97E2C5D52.png.
- This is a 148 pixels wide and 233 pixels high PNG file.
- This picture has 4486 colors, so JPEG is a better choice.
- Success: Old: 29880, New: 5642 (-82%). We'll use the new picture.
Patching content.xml with new image file name.
Patching styles.xml with new image file name.
Patching manifest.xml with new image file name.
Optimizing Pictures/10000201000003E3000003E4CFFA65E3.png.
- This is a 995 pixels wide and 996 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 196597, New: 624633. We better keep the original.
Optimizing Pictures/100002010000013C0000021EDE4EFBD7.png.
- This is a 316 pixels wide and 542 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 159495, New: 224216. We better keep the original.
Optimizing Pictures/10000200000002120000014A19C2D0EB.gif.
- This is a 530 pixels wide and 330 pixels high GIF file.
- This image is transparent. Can't convert to JPEG.
- We will convert this image to PNG, which is probably more efficient.
- Failure: Old: 39821, New: 56736. We better keep the original.
Optimizing Pictures/100000000000020D0000025EB55F72E3.png.
- This is a 525 pixels wide and 606 pixels high PNG file.
- This picture has 17123 colors, so JPEG is a better choice.
- Success: Old: 146544, New: 16210 (-89%). We'll use the new picture.
Patching content.xml with new image file name.
Patching styles.xml with new image file name.
Patching manifest.xml with new image file name.
Optimizing Pictures/10000000000000200000002000309F1C.png.
- This is a 32 pixels wide and 32 pixels high PNG file.
- This picture has 256 colors, so JPEG is a better choice.
- Success: Old: 859, New: 289 (-67%). We'll use the new picture.
Patching content.xml with new image file name.
Patching styles.xml with new image file name.
Patching manifest.xml with new image file name.
Optimizing Pictures/10000201000001BB0000006B7305D02E.png.
- This is a 443 pixels wide and 107 pixels high PNG file.
- This image is transparent. Can't convert to JPEG.
- We will try re-encoding this image with PNG compression level 9.
- Failure: Old: 730, New: 24071. We better keep the original.
All images optimized.
Re-packing...
Success: The new file is only 67% as big as the original!
Cleaning up...
Done.Neat. We just shaved a third off of a 1.3MB presentation file and it still looks as good as the original!
As you can see, the script goes through each image one by one and tries to come up with better ways of encoding images. The basic rules are:
- If an image if PNG or GIF and it has more than 128 colors, it's probably better to convert it to JPEG (if it doesn't use transparency). It also tries recompressing GIFs and other legacy formats as PNGs if JPEG is not an option.
- Images bigger than 1280x1024 don't make a lot of sense in a presentation, so they're resized to be at most that size.
- JPEG allows to set a quality level. 75% is "good enough" for presentation purposes, so we'll try that and see how much it buys us.
- If an image if PNG or GIF and it has more than 128 colors, it's probably better to convert it to JPEG (if it doesn't use transparency). It also tries recompressing GIFs and other legacy formats as PNGs if JPEG is not an option.
Before I give you the script, here's the obvious
Disclaimer: Use this script at your own risk. Always check the shrunk presentation for any errors that the script may have introduced. It only works 9 out of 10 times (sometimes, there's some funkiness about how OpenOffice uses images going on that I still don't understand...), so you have to check if it didn't damage your file.
The script works with Solaris (of course), but it should also work in any Linux or any other Unix just fine. It relies on ImageMagick to do the image heavy lifting, so make sure you have identify(9E) and convert(9E) in your path.
My 22 MB Systems Overview presentation was successfully shrunk into a 13MB one, so I'm happy to report that after so many years, this little script is still very useful. I hope it helps you too, let me know how you use it and what shrink-ratios you have experienced!
"Shrink big presentations with ooshrink" has been brought to you by Constantin's Blooog.
This entry was created on 2007-11-27 03:20:08.0 PST and is associated with the following tags:
imagemagick
imaging
open
openoffice
opensource
script
source
staroffice
useful
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Comments [12]
Sunday October 21, 2007 How to burn high resolution DVD-Audio DVDs on Solaris and Linux (And it's legal!)
This weekend I've burned my first DVD-Audio DVD with high resolution music at 96 kHz/24 Bit.
It all started with this email I got from Linn Records, advertising the release of their Super Audio Surround Collection Vol 3 Sampler (Yes, targeted advertising works, but only if customers choose to receive it), which is offered in studio master quality FLAC format files, as a download. Gerald and I applauded Linn Records a few months ago for offering high quality music as lossless quality downloads, so I decided to try out their high resolution studio master quality offerings.
The music comes as 96kHz/24 Bit FLAC encoded files. These can be played back quite easily on a computer with a high resolution capable sound card, but computers don't really look good in living rooms, despite all the home theater PC and other efforts. The better alternative is to burn your own DVD-Audio and then use a DVD-A capable DVD player connected to your HiFi-amplifier to play back the music.
There's a common misconception that "DVD-Audio" means "DVD-Video" without the picture which is wrong. DVD-Video is one standard, aimed at reproducing movies, that uses PCM, AC-3, DTS or MP2 (mostly lossy) for encoding audio, while DVD-Audio sacrifices moving pictures (allowing only still ones for illustration) so it can use the extra bandwidth for high resolution audio, encoded as lossless PCM or lossless MLP bitstreams. Also, note that it is not common for regular DVD-players to accept DVD-Audio discs, they must state that they can handle the format, otherwise you're out of luck. Some if not most DVD-Audio Discs are hybrid in that they offer the content stored in DVD-Audio format additionally as DVD-Video streams with one of the lossy DVD-Video audio codecs so they can be played on both DVD-Video and DVD-Audio players.
Now, after having downloaded a bunch of high-res FLAC audio files, how can you create a DVD-Audio disc? Here's a small open source program called dvda-author that does just that: Give it a bunch of FLAC or WAV files and a directory, and it'll create the correct DVD-A UDF file structure for you. It compiles very easily on Solaris so I was able to use my Solaris fileserver in the basement where I downloaded the songs to. Then you give the dvda-author output directory along with a special sort file (supplied by dvda-author) to mkisofs (which is included in Solaris in the /usr/sfw directory) and it'll create a DVD ISO image that you can burn onto any regular DVD raw media. It's all described nicely on the dvda-author How-To page. Linn Records also supplies a PNG image to download along with the music that you can print and use as your DVD-Audio cover.
And how about iPods and other MP3-Players? Most open source media players such as the VideoLan Client (VLC) can transcode from high resolution FLAC format to MP3 or AAC so that's easily done, too. For Mac users, there's a comfortable utility called XLD that does the transcoding for you.
Here's common misconception #2: Many people think AAC is proprietary to Apple, mostly because Apple is heavily advertising its use as their standard for music encoding. This is wrong. AAC is actually an open standard, it is part of the ISO/IEC MPEG-4 specification and it is therefore the legitimate successor to MP3. AAC delivers better audio quality at lower bitrates and even the inventors of MP3, the Fraunhofer IIS institute treat AAC as the legitimitate successor, just check their current projects page under the letter "A". Apple developed the "Fairplay" DRM extension to Quicktime (which is the official MPEG-4/AAC encapsulation format) to be able to sell their iTunes Music Store as a download portal to the music industry. Fairplay is proprietary to Apple, but has nothing to do with AAC per se.
As much as I love Apple's way of using open standards wherever possible, I don't think it's a good thing that their marketing department creates the illusion of these technologies being Apple's own. This is actually an example of how AAC suffers in the public perception because people think it's proprietary where the opposite is true.
How is the actual music, you ask? Good. The album is a nice mixture of jazz and classical music, both in smooth and in more lively forms, great for a nice dinner and produced with a very high quality. Being a sampler, this album gives you a good overview of current Linn Records productions, so you can choose your favourite artists and then dig deeper into the music you liked most.
There's one drawback still: The high-res files available on the Linn Records download store are currently stereo only, while the physical SACD releases come with 5.1 surround sound. It would be nice if they could introduce 5.1 FLAC downloads in the future. That would make downloading high resolution audio content perfect, and this silly SACD/DVD-Audio/Dolby-TrueHD/DTS-HD Master Audio war would finally be over.
P.S.: A big hello to the folks at avsforum.com who were so kind to link to my previous high resolution audio entry!
"How to burn high resolution DVD-Audio DVDs on Solaris and Linux (And it's legal!)" has been brought to you by Constantin's Blooog.
This entry was created on 2007-10-21 04:45:44.0 PST and is associated with the following tags:
audio
diy
dvd
dvd-a
dvda
hd-audio
howto
open
opensource
solaris
source
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Tuesday October 16, 2007 Walking through the CEC 2007 JavaFX Message Prompter Source
Now that I'm back from CEC and out of jetlag, I've had some time to clean up the CEC 2007 Message Prompter source code. Thanks to all those who asked for it, that was quite a motivation.
The CEC Message Prompter source code is free for your reading pleasure under an as-is basis, no warranty, no support, etc. Still, comments are of course very welcome.
The easiest way to try this out is to load up NetBeans (I use the current Beta 6), install the JavaFX module, then create a new JavaFX project. The stuff in the source code archive goes into the src subdirectory of your new JavaFX project. Choose "Main.fx" as the main class and feel free to enable Java Web Start.
In order to compile/run the app, you also need JAXB 2.0 (or use J2SE 6) and the mySQL JDBC Connector installed in NetBeans as libraries and assigned to the project you use for this app.
After starting the app, you'll see the window above. To the top is the message source selection GUI. Choose whether you want to have a database or a URL (for XML) connection. A sample XML file with some messages is included, so you probably want to use the URL method. Enter the file URL where you have your messages stored into the URL field, then click on the right (next) or left (previous) or the X (clear) buttons to display the messages. The optional Session field is for filtering messages by session ID but we never got to use it yet.
Before I start with the code, a few words of introduction: This is my first JavaFX project and I welcome any suggestions on how to better code in JavaFX. It is also my first Java/NetBeans project since a long time, so I'm sure I can still learn a lot more about how to properly do it. But the learning journey into creating this app has been a fun and instructive one, so I hope this code can help others learn more about JavaFX too. If I had to do it again (And I hope I will, next year), I'd do some stuff differently, which I'll discuss at the end of this posting.
Let's walk through the code in roughly the order of how the message flow works:
- The basic idea is this:
- The audience sends their questions, feedback, messages etc. to the CEC backstage team through either Email, Instant Messaging or SMS through special Email or IM accounts or mobile phone numbers. The CEC backstage team reads the messages and stores them in a database where they can be approved, marked for deletion, marked for display on the Message Prompter and assigned a sequence to display in.
- The CEC Message Prompter is the application that the people on stage and occasionally the audience see/s and where the current question to be asked to the people on stage is displayed. So the app has to fetch messages from the database and display them on screen on demand and in a visually intuitive way.
- For testing/development/backup purposes, the Message Prompter can also accept messages out of a single XML file instead of a database.
- The top level directory is supposed to go into the src subdirectory of a NetBeans JavaFX project, but I guess you could as well include this easily into any other IDE or just work from the command-line out of this directory. CECMessage.xsd is an XML schema courtesy of Simon Cook that defines the XML format for a list of messages. ExampleMessages.xml contains a bunch of messages for testing purposes. Most of the source code is in the cecmessaging subdirectory which is the name of the Java package bundle for this app. If you apply the Message schema to the JAXB xjc.sh script, it creates the java classes in the org/netbeans/xml/schema/cecmessage directory which describe the corresponding java objects.
- Some things are best left to real Java, in this case the message fetching and conversion into JavaFX digestable classes. The nice thing about JavaFX is that it can seamlessly use Java classes for doing any heavy-lifting. Messages can come in as an XML file or from a database, in both cases they are fetched from Java helper classes that handle the complexity of fetching messages and who return an object structure to the main JavaFX application.
In the case of messages coming in as a single XML file, the file is parsed by the XMLHelper class in cecmessaging/XMLHelper.java using the JAXB framework. The resulting object structure can then be interpreted by the JavaFX main class. Make sure you include JAXB 2.0 or later if you use J2SE 5, in J2SE 6 it's already included.
If messages are to be retrieved from a database, then the DBHelper class in cecmessaging/DBHelper.java is used. It uses the mySQL JDBC connector for database access but you could easily plug in any other database connector. For simplicity, the database data is converted into a JAXB structure as if it was coming out of an XML document. Here is the definition of the database that Simon created:
Database: cec Table: message
Both XMLHelper and DBHelper sort the messages by the displayOrder field, then by id. The sort comparator for this particular ordering is in CECMessageComparator.java.Field Type id int(11) sender varchar(100) message text topic varchar(100) timeSent timestamp session_id int(11) device varchar(10) approved tinyint(1) deleted tinyint(1) to_be_asked tinyint(1) display_sort_order int(11) - The heart of the Message Prompter lives in cecmessaging/Main.fx.
- It starts with a few data structures:
- The AnimValues class stores font sizes, colors, duration and other parameters used for animation. JavaFX does not let you specify defaults as part of the class definition, hence the attribute commands.
- The Message class is modeled after the corresponding JAXB CECMessage class. It adds a few attributed to track a particular message's color and font size. The font size and color of a message depends on its position (whether it is highlighted or not) and can change while it is animated during transitions. That's why we need to keep track of them. The alive attribute is not used right now, it may become useful if I rework the animation stuff.
- The Tag class is for handling, well, tags. Every word that shows up as a message, author, device or topic is treated as a tag and a tag cloud is generated based on how often the word shows up on the screen. This class stores the tag word, counts the number of appearance and stores the current font size of that tag on screen. Again, we need to track font size for animation.
- The MessageList class is the main model class the application uses. It contains an AnimValue class, a list of Message class messages and list of tags. It knows where the messages come from and where the original message data in JAXB format is. It keeps track of the GUI Labels that graphically represent the messages on screen plus it knows how many messages to display at once, which one is to be highlighted and other useful parameters.
- The following operation statements are really methods. They are written in script-like manner rather than in object-oriented manner. This means that they are not associated to a particular class other than the main one. Next time, I might use a more strict object-oriented approach, but hey, this is a scripting language, isn't it?
- MessageSignature computes a single string out of all fields in a message for comparison purposes. Somehow .equals or .toString didn't work for me as expected, so Implemented this simple mechanism to see if two messages are equal.
- ClearMessages clears all messages, its associated Label objects and makes sure that dying messages are animated and their tags updated. Actually, today the death of a message isn't animated yet but I hope to implement a nice way of dying for messages. I loved my Apple Newton back then in the 90ies and it had this nice animation where deleted stuff would vanish in a puff of smoke :).
- CecmToMessage takes a JAXB message created by the XMLHelper or DBHelper class and creates the corresponding JavaFX Message instance. It also handles basic true/false associations for the approved and deleted fields, which are meant to be boolean but are actually strings in the XML schema.
- MessageToLabel creates a Java Swing Label object that displays the message on screen. The nice thing about the JavaFX Label implementation is that it understands HTML. So we can use HTML freely here to control how the message is to be seen. Notice the bind statement where the Label text is: It ties the Label's text content to the Message's attributes (color, size, content). This means that whenever any of these attributes are changed in the Message object, the corresponding Label object is changed as well! This is a very nice mechanism for Model-View-Controller like programming and a big time saver when coding.
- The messageDisplayable function decides whether a message is supposed to be displayed. This is just a logic expression checking the approved, deleted and toBeAsked fields and filtering by sessionId (In case one wants to restrict messages to a particular session). One could have implemented the filtering at the XMLHelper or at the DBHelper level, but I felt it would be better to have full control over displaying messages from the app itself.
- UpdateMessages checks all currently displayed messages against their counterparts in the XML file or the DB. The idea here is that we want to be able to change a message even if it's already displayed in the application (you know, when accidentally a bad word came through :) ). This is called regularly before adding new messages to the screen.
- compareMessageOrder does just that. Messages come in already sorted, but we still need to decide on ordering when going through them to detect whether a message is missing etc. (The naming is wrong, it should start uppercase. This is because this operation started as a function but then if-then is not accepted in functions by JavaFX...).
- NextMessage adds a message to the display list. It also deals with the unexpected complexity of deciding which message to highlight in certain corner cases. For instance, when we want to preview 2 messages, the third one is to be highlighted, but if you only have 0-2 messages on screen, the highlight should be on the last etc. When done, the message animator is called to animate any newly higlighted or unhighlighted messages and the tags are updated.
- PreviousMessage does the opporsite of NextMessage. Again, the handling of the highlight is a tad more complex than I would have wanted it. Again, we animate here as well.
- RefreshTags goes through all messages displayed on screen and makes sure the tag list is up to date. Then it starts animation for those tags that have changed.
- AnimateMessages checks all messages and whether their font sizes match their position and highlighting status. Then, it animates all messages that have changed their status into their destination sizes and colors. Animation is handled through the dur operator. It assigns a list of values to a single parameter in sequence, during a specified time. So when we want a piece of text to grow from 0 to 20 pixels in size during 500 milliseconds, we say something like text.size = [0..20] dur 500. Very nice! Color animations work by applying a string array with the color values in sequence to a string variable. I wasn't confident on how the animation works in terms of concurrency (for instance, if another thread happens to change a value while it is animated) and I've seen cases where the font sizes weren't correct (and that cost me quite some sweat drops!) so I added some watchdog code to make sure the font size is correct after the end of the animation. Now that I've seen the CEC 2007 JavaFX session (sic!), I know a bit more about how this is supposed to work so hopefully I won't need it any more :).
- AnimateTags does similar things to the tags, a tad easier to do.
- The LoadProperties stuff is not used at the moment, so isn't the properties file included with the source. I was planning to outsource all relevant defaults and constants into an external properties file, but didn't have the time to do it. But here's a start...
- The Main part is fairly straight forward: It first instantiates the MessageList model structure with some default values, then proceeds to instantiate the GUI elements. Another nice thing about JavaFX is the declarative syntax where you just write down the GUI class with all the desired values and the runtime system takes care of instantiating the classes, hooking them together and assigning values to them, as well as tying in the methods to be called when a GUI element is activated. Also, the bind command is your best friend here in that it automatically binds GUI attributes to the model classes and saves you the hassle of implementing callback methods etc. You don't even need a GUI builder, just write down the widget hierachy and you're done. Very convenient.
That was it. All in all, learning JavaFX was a fun experience. And you can do it too, just go to the OpenJFX website and check out the tutorials and references.
What would I do differently if I had to write this app from scratch? Probably one or more of the following:
- Use real object oriented style by attaching methods to classes etc. Possibly different classes in different files, loosely coupled by the main class, as in this nice Mariah Carey website example.
- Rework the animation so it works on triggers. Triggers are a way of coupling code to variables, similar to binding. So, whenever a variable is changed, the trigger code gets executed. For instance, the tags could be updated and animated using triggers.
- Introduce more eye-candy. JavaFX comes with full Java2D support, so I'd dig in deeper into its classes to implement nicer animations.
- Make it more interactive by letting GUI elements slide in and out only when necessary so there's more real estate for the messages.
- Introduce images and symbols to help with the eye-candyness.
Thank you for reading this and I hope you enjoyed this JavaFX example. Let me know your thoughts by using the comment function or by sending me email!
"Walking through the CEC 2007 JavaFX Message Prompter Source" has been brought to you by Constantin's Blooog.
This entry was created on 2007-10-16 03:44:16.0 PST and is associated with the following tags:
2007
cec
cec2007
client
java
javafx
jdbc
messaging
netbeans
open
opensource
source
suncec2007
swing
xml
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Thursday September 06, 2007 7 Easy Tips for ZFS Starters
So you're now curious about ZFS. Maybe you read Jonathan's latest blog entry on ZFS or you've followed some other buzz on the Solaris ZFS file system or maybe you saw a friend using it. Now it's time for you to try it out yourself. It's easy and here are seven tips to get you started quickly and effortlessly:
1. Check out what Solaris ZFS can do for you
First, try to compose yourself a picture of what the Solaris ZFS filesystem is, what features it has and how it can work to your advantage. Check out the CSI:Munich video for a fun demo on how Solaris ZFS can turn 12 cheap USB memory sticks into highly available, enterprise-class, robust storage. Of course, what works with USB sticks also works with your own harddisks or any other storage device. Also, there are great ZFS screencasts that show you some more powerful features in an easy to follow way. Finally, there's a nice writeup on "What is ZFS?" at the OpenSolaris ZFS Community's homepage.
2. Read some (easy) documentation
It's easy to configure Solaris ZFS. Really. You just need to know two commands: zpool (1M) and zfs (1M). That's it. So, get your hands onto a Solaris system (or download and install it for free) and take a look at those manpages. If you still want more, then there's of course the ZFS Administration Guide with detailed planning, configuration and troubleshooting steps. If you want to learn even more, check out the OpenSolaris ZFS Community Links page. German-speaking readers are invited to read my german white paper on ZFS or listen to episode #006 of the POFACS podcast.
3. Dive into the pool
Solaris ZFS manages your storage devices in pools. Pools are a convenient way of abstracting storage hardware and turning it into a repository of blocks to store your data in. Each pool takes a number of devices and applies an availability scheme (or none) to it. Pools can then be easily expanded by adding more disks to them. Use pools to manage your hardware and its availability properties. You could create a mirrored pool for data that should be protected against disk failure and that needs fast access to hardware. Then, you could add another pool using RAID-Z (which is similar, but better than RAID-5) for data that needs to be protected but where performance is not the first priority. For scratch, test or demo data, a pool without any RAID scheme is ok, too. Pools are easily created:
zpool create mypool mirror c0d0 c1d0
Will create a mirror out of the two disk devices c0d0 and c1d0. Similarly, you can easily create a RAID-Z pool by saying:
zpool create mypool raidz c0d0 c1d0 c2d0
The easiest way to turn a disk into a pool is:
zpool create mypool c0d0
It's that easy. All the complexity of finding, sanity-checking, labeling, formatting and managing disks is hidden behind this simple command.
If you don't have any spare disks to try this out with, then you can just create yourself some files, then use them as if they were block devices:
# mkfile 128m /export/stuff/disk1
# mkfile 128m /export/stuff/disk2
# zpool create testpool mirror /export/stuff/disk1 /export/stuff/disk2
# zpool status testpool
pool: testpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
testpool ONLINE 0 0 0
mirror ONLINE 0 0 0
/export/stuff/disk1 ONLINE 0 0 0
/export/stuff/disk2 ONLINE 0 0 0
errors: No known data errors
The cool thing about this procedure is that you can create as many virtual disks as you like and then test ZFS's features such as data integrity, self-healing, hot spares, RAID-Z and RAID-Z2 etc. without having to find any free disks.
When creating a pool for production data, think about redundancy. There are three basic properties to storage: availability, performance and space. And it's a good idea to prioritize them in that order: Make sure you have redundancy (mirroring, RAID-Z, RAID-Z2) so ZFS can self-heal data when stuff goes wrong at the hardware level. Then decide how much performance you want. Generally, mirroring is faster and more flexible than RAID-Z/Z2, especially if the pool is degraded and ZFS needs to reconstruct data. Space is the cheapest of all three, so don't be greedy and try to give priority to the other two. Richard Elling has some great recommendations on RAID, space and MTTDL. Roch has also posted a great article on mirroring vs. RAID-Z.
4. The power to give
Once you have set up your basic pool, you can already access your new ZFS file system: Your pool has been automatically mounted for you in the root directory. If you followed the examples above, then you can just cd to /mypool and start using ZFS!
But there's more: Creating additional ZFS file systems that use your pool's resources is very easy, just say something like:
zfs create mypool/home
zfs create mypool/home/johndoe
zfs create mypool/home/janedoe
Each of these commands only takes seconds to complete and every time you will get a full new file system, already set up and mounted for you to start using it immediately. Notice that you can manage your ZFS filesystems hierarchically as seen above. Use pools to manage storage properties at the hardware level, use filesystems to present storage to your users and applications. Filesystems have properties (compression, quotas, reservations, etc.) that you can easily administer using zfs set and that are inherited across the hierarchy. Check out Chris Gerhard's blog on more thoughts about file system organization.
5. Snapshot early, snapshot often
ZFS snapshots are quick, easy and cheap. Much cheaper than the horrible experience when you realize that you just deleted a very important file that hasn't been backed up yet! So, use snapshots whenever you can. If you think about whether to snapshot or not, just do it. I recently spent only about $220 on two 320 GB USB disks for my home server to expand my pool with. At these prices, the time you spend thinking about whether to snapshot or not may be more worth than just buying more disk.
Again, Chris has some wisdom on this topic in his ZFS snapshot massacre blog entry. He once had over 60000 snapshots and he's snapshotting filesystems by the minute! Since snapshots in ZFS “just work” and since they only take up the space that actually changes between snapshots, there's really no reason to not doing snapshots all the time. Maybe once per minute is a little bit exaggerated, but once a week, once per day or once an hour per active filesystem is definitely good advice.
Instead of time based snapshotting, Chris came up with the idea to snapshot a file system shared with Samba whenever the Samba user logs in!
6. See the Synergy
ZFS by itself is very powerful. But the full beauty of it can be unleashed by combining ZFS with other great Solaris 10 features. Here are some examples:
Tim Foster has written a great SMF service that will snapshot your ZFS filesystems on a regular basis. It's fully automatic, configurable and integrated with SMF in a beautiful way.
-
ZFS can create block devices, too. They are called zvols. Since Nevada build 54, they are fully integrated into the Solaris iSCSI infrastructure. See Ben Rockwood's blog entry on the beauty of iSCSI with ZFS.
-
A couple of people are now elevating this concept even further: Take two Thumpers, create big zvols inside them, export them through iSCSI and mirror over them with ZFS on a server. You'll get a huge, distributed storage subsystem that can be easily exported and imported on a regular network. A poor man's SAN and a powerful shared storage for future HA clusters thanks to ZFS, iSCSI and Thumper! Jörg Möllenkamp is taking this concept a bit further by thinking about ZFS, iSCSI, Thumper and SAM-FS.
-
Check out some cool Sun StorageTek Availability Suite and ZFS demos here.
ZFS and boot support is still in the works, but if you're brave, you can try it out with the newer Solaris Nevada distributions on x64 systems. Think about the possibilities together with Solaris Live Upgrade! Create a new boot environment in seconds while not needing to find or dedicate a new partition, thanks to snapshots, while saving most of the needed disk space!
And that's only the beginning. As ZFS becomes more and more adopted, we'll see many more creative uses of ZFS with other Solaris 10 technologies and other OSes.
7. Beam me up, ZFS!
One of the most amazing
features of ZFS is zfs send/receive.
zfs send
will turn a ZFS filesystem into a bitstream that you can save to a
file, pipe through bzip2 for compression or send through ssh to a
distant server for archiving or for remote replication through the
corresponding zfs
receive command.
It also supports incremental sending and receiving out of subsequent
snapshots through the -i
modifier.
This is a powerful feature with a lot of uses:
-
Create your Solaris zone as a ZFS filesystem, complete with applications, configuration, automation scripts, users etc.,
zfs send | bzip2 >zone_archive.zfs.bz2it for later use. Then, unpack and create hundreds of cloned zones out of this master copy. -
Easily migrate ZFS filesystems between pools on the same machine or on distant machines (through ssh) with
zfs send/receive. -
Create a crontab entry that takes a snapshot every minute, then
zfs send -iit over ssh to a second machine where it is piped intozfs receive. Tadah! You'll get free, finely-grained, online remote replication of your precious data. -
Easily create efficient full or incremental backups of home directories (each in their own ZFS filesystems) through ZFS send. Again, you can compress them and treat them like you would, say, treat a tar archive.
See? It is easy, isn't it? I hope this guide helps you find your way around the world of ZFS. If you want more, drop by the OpenSolaris ZFS Community, we have a mailing list/forum where bright and friendly people hang out that will be glad to help you.
"7 Easy Tips for ZFS Starters" has been brought to you by Constantin's Blooog.
This entry was created on 2007-09-06 11:20:15.0 PST and is associated with the following tags:
adoption
community
data
filesystem
free
howto
innovation
introduction
open
opensolaris
opensource
software
solaris
storage
tips
unix
zfs
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Thursday August 23, 2007 Cool Apple-Like Photo Animations With POV-Ray, ImageMagick and Solaris
One of the features I like most about Apple's iPhoto application is the transition effect in slideshows where the whole screen becomes a cube that rotates into the next photograph. The same effect is also used when switching users, etc.
Recently we took a team photograph for an internal web page. I wanted that effect and I love the open source raytracer POV-Ray so I wrote a script that renders the same animation effect and creates an animated GIF using ImageMagick. You can see an example output to the right featuring photos of some popular Sun products. BTW, check out photos.sun.com for free, high-quality access to Sun product photography.
To create your own photocubes, you just need POV-Ray and ImageMagick in your path and the photocube.sh script. Being open source, all run on Solaris but also on Linux, NetBSD or any other operating system that can run open source software. I'd love to try this script out on a Niagara 2 system with its 8 cores, 16 pipelines, 64 threads and 8 FPUs. Hmmm, all rendering frames in parallel :).
There are already precompiled distributions of POVRay and ImageMagick on Blastwave that you can install very easily onto your Solaris machine if you don't have them already.
Just call the script with 6 URLs or pathnames. It will then automatically read in the images, render the animation frames and then combine them all into an animated GIF:
-bash-3.00$ ../photocube.sh *.jpg
Converting images to be quadratic...
Fetching and processing 721_DRIVE-open.1024x768.jpg
Fetching and processing 773_FrtAbv-78L-PWR-FAN.1024x768.jpg
Fetching and processing 915_Lside-plexiOFF.1024x768.jpg
Fetching and processing IMG_4551.1024x768.jpg
Fetching and processing blackbox_wind_turbine.1024x768.jpg
Fetching and processing ultra_cool_combo.1024x768.jpg
Rendering animation frames...
Creating animated gif file...
-bash-3.00$ ls
721_DRIVE-open.1024x768.jpg blackbox_wind_turbine.1024x768.jpg
773_FrtAbv-78L-PWR-FAN.1024x768.jpg photocube.gif
915_Lside-plexiOFF.1024x768.jpg ultra_cool_combo.1024x768.jpg
IMG_4551.1024x768.jpg
-bash-3.00$
The script uses ImageMagick to make the pictures quadratic and to limit their size to 1200x1200 pictures if necessary. Since the -extent switch is fairly new, you'll need to use a newer distribution of ImageMagick, the one on the Solaris Companion CD is sadly not new enough. The POVRay source (embedded in the script) uses an S-curve to turn the cube which produces a natural and smooth acceleration/decelleration effect. It would have been more efficient to let POVRay output PNG files rather than TGA but for some reason some of the PNG files that POVRay produces were not compatible with ImageMagick.
Feel free to modify this script to your needs. You may want to experiment with other ways of animating the cube or other image transition effects. Maybe you want to use ffmpeg to create real video files instead of animated GIFs. Be careful when cranking up the number of frames while using ImageMagick to create animated GIFs, ImageMagick wants to suck in all frames into memory before creating the animated GIF and so you may end up using a lot of memory. If someone has a more elegant, scriptable animated GIF creator, please leave me a comment.
I hope you enjoy this little exercise in raytracing and animation. Let me know if you have suggestions or other ideas to improve this script!
"Cool Apple-Like Photo Animations With POV-Ray, ImageMagick and Solaris" has been brought to you by Constantin's Blooog.
This entry was created on 2007-08-23 13:20:47.0 PST and is associated with the following tags:
apple
cool
diy
howto
imagemagick
iphoto
opensolaris
opensource
photos
pov-ray
raytracing
script
slideshow
solaris
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Comments [4]
Thursday August 16, 2007 ZFS Snapshot Replication Script
One of the OpenSolaris' ZFS filesystem's greatest features are its snapshots. You can easily create a snapshot by saying zfs create pool/filesystem@snapshot and then replicate that snapshot by saying something like zfs send pool/filesystem@snapshot | zfs receive newpool/some_other_filesystem. So far, so great.
Now let's say you have a nice pool and have been creating snapshots on a regular basis. After a few months, you decide to remodel your pool layout or migrate some of your filesystems over to a new pool for whatever reason. Then, you're facing a lot of those zfs send and receive commands. Especially, if you're like Chris and do snapshots on a per-minute basis. That's why the ZFS community is now waiting for 6421958: A zfs send -r option.
I had to migrate quite a few filesystems and many snapshots (thanks to Tim's excellent ZFS Snapshot SMF Service) lately when I set up a new pool strategy for my home server so I wrote myself a script to do the replication job. Since it may take some time for the send -r feature to be implemented, I turned it into a more generic utility.
Disclaimer: Please be advised that this script has only been tested a couple of times and it is provided to you completely on an "as-is" basis. Please have a look at the script to understand how it works and try it out on some non-risky pools and filesystems before you do real stuff with it. Run a backup before using this script and don't shoot me if something goes wrong.
Ok, what can this script do for you? First of all, check out its -h flag to see what options it provides:
# zfs-replicate -h
usage: zfs-replicate [-h] [-F ] [-n] [-s] [-v]
[-m [-c "FMRI|pattern[ FMRI|pattern]...]" ]
source dest
where source and dest is a ZFS filesystem, snapshot or volume.
Options:
-h: Print this help.
-F: Force a rollback of the destination filesystem to the most recent
snapshot before replicating a snapshot. This is equivalent to the -F
option of zfs receive.
-n: Don't actually replicate anything, just print what would be done.
This will only print the next step but nothing dependent on that step
since it won't actually be executed. For instance, -ns will print the
snapshot command but not print the subsequent send/receive command as it
depends on the snapshot actually being taken.
-s: After sending existing snapshots, make a final one and replicate it as well.
This option requires that the source be a filesystem and not a snapshot.
-m: After sending all snapshots, migrate the source to the dest filesystem by
unmounting the source filesystem and changing the new filesystem's
mountpoint to that of the source one. This option includes -s.
-c: A space delimited list of SMF services in quotes to be temporarily disabled
before unmounting the source, then re-enable after changing the mountpoint
of the destination. Requires -m.
-v: Be verbose.Great, let's try it out. Here's a pool with some data and some snapshots as well as another, empty pool:
# zfs list -r piscina
NAME USED AVAIL REFER MOUNTPOINT
piscina 384M 87.6M 19K /piscina
piscina/ficheros 384M 87.6M 384M /piscina/ficheros
piscina/ficheros@instantanea1 17K - 128M -
piscina/ficheros@instantanea2 17K - 256M -
piscina/ficheros@instantanea3 0 - 384M -
# zfs list -r lago
NAME USED AVAIL REFER MOUNTPOINT
lago 105K 472M 18K /lago
Now, let's copy the ficheros filesystem along with all its snapshots in one go:
# zfs-replicate -v piscina/ficheros lago
Sending piscina/ficheros@instantanea1 to lago.
Sending piscina/ficheros@instantanea2 to lago.
(incremental to piscina/ficheros@instantanea1.)
Sending piscina/ficheros@instantanea3 to lago.
(incremental to piscina/ficheros@instantanea2.)
# zfs list -r lago
NAME USED AVAIL REFER MOUNTPOINT
lago 384M 87.6M 20K /lago
lago/ficheros 384M 87.6M 384M /lago/ficheros
lago/ficheros@instantanea1 17K - 128M -
lago/ficheros@instantanea2 17K - 256M -
lago/ficheros@instantanea3 0 - 384M -
It works. And it automatically used incremental snapshots as well to save space, too!
If we now add another snapshot to our original pool piscina and then run zfs-replicate again, it will skip already replicated snapshots and just copy those that are additional:
# zfs snapshot piscina/ficheros@instantanea4
# zfs-replicate -v piscina/ficheros lago
Sending piscina/ficheros@instantanea4 to lago.
(incremental to piscina/ficheros@instantanea3.)
# zfs list -r lago
NAME USED AVAIL REFER MOUNTPOINT
lago 384M 87.6M 20K /lago
lago/ficheros 384M 87.6M 384M /lago/ficheros
lago/ficheros@instantanea1 17K - 128M -
lago/ficheros@instantanea2 17K - 256M -
lago/ficheros@instantanea3 0 - 384M -
lago/ficheros@instantanea4 0 - 384M -
This is useful because you can now run this script on regularly basis to have one pool automatically backed up to another pool. In fact, the -s option will first make sure all existing snapshots are copied over, then create a new snapshot for you and copy that one over as well, all in one command.
Sometimes, the destination filesystem gets touched, or otherwise acted upon and then zfs receive to it will no longer work. In that case, you can use the -F switch which will force a rollback of the destination filesystem to its latest snapshot and then you'll be back in business.
Finally, another scenario is file system migration: You have a filesystem in one pool and want to migrate it with all it's snapshots to another pool, with minimal downtime. This can be done using the -m option: First copy all existing snapshots while the source file system is still live, then unmount it, create a final snapshot, copy that one over to the destination file system as well, set the destination filesystem's mountpoint to be the same as the source filesystem's one and remount the destination file system. (Please note that the script might appear frozen when unmounting while it's waiting for some open files to be closed, etc.).
If you're worried about some daemons depending on your filesystem's availability (like Samba), you can use the -c option to provide their names. zfs-replicate will then bring down the matching SMF services right before unmounting and restart them automatically after re-mounting the migrated filesystem. Again, you might need to wait until the SMF service is really down (Read: The last Samba connection has closed).
I hope this script is useful to you and again, I assume you know what you're doing and do some testing before using it in production. I'm sure there are still some bugs and shortcomings so please send me email to constantin (dot) gonzalez (at) sun (dot) com or leave a comment and I'll try to make the script better for you.
Many thanks to Chris Gerhard, whose backup script was an inspiration for me in hacking together this utility. Also, many thanks to Tim Foster for some code-review and initial feedback (Sorry, I haven't managed to implement some locking yet...). Let me know when you're in Munich and you'll get some well-deserved beer!
"ZFS Snapshot Replication Script" has been brought to you by Constantin's Blooog.
This entry was created on 2007-08-16 13:41:02.0 PST and is associated with the following tags:
administration
filesystem
howto
open
opensolaris
opensource
programming
replication
script
shell
snapshot
software
solaris
source
unix
utility
zfs
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Sunday August 12, 2007 ZFS Interview in the POFACS Podcast (German)
Last week, I've been interviewed by the german podcast POFACS, the podcast for alternative computer systems. Today, the interview went live, so if you happen to understand the german language and want to learn about ZFS while driving to work or while jogging, you're invited to listen to the interview.
I was actually amazed at how long the interview turned out: It's 40 minutes, while recording the piece only felt like 20 minutes or so. The average commute time in germany is about 20 minutes, so this interview will easily cover both ways to and from work. But there's more: This episode of POFACS also introduces you to the NetBSD operating system, the German Unix User Group GUUG. Finally, the guys at POFACS were also so kind to feature the HELDENFunk podcast in a short introductory interview. Thanks!
So with a total playing time if 1 hour and 20 minutes, this episode has you covered for at least two commutes or a couple of jogging runs :).
"ZFS Interview in the POFACS Podcast (German)" has been brought to you by Constantin's Blooog.
This entry was created on 2007-08-12 10:41:54.0 PST and is associated with the following tags:
community
filesystem
free
german
guug
open
opensolaris
opensource
podcast
podcasting
pofacs
solaris
technology
zfs
You're welcome to use this Permalink
, add a comment below or send your feedback to constantin at sun dot com.
Main | Next page »
