The thing about trees ... and my Kindle DX

Friday Jul 03, 2009

I can summarize my thoughts on trees pretty succinctly, I love a good tree. And with that in mind I finally made the leap and bought my Amazon Kindle DX.

I can safely say that I am holding onto a little bit of the future, it is incredible. The things that drove me over the top to get the Kindle are:

• I am VERY schizophrenic in my reading enjoyment, sometimes having as many as 4 books for reading open, 2 or 3 technical documents, and several large PDFs full of APIs
  
• I received The Denver Post Thursdays through Sundays and didn't read it most of those days ... but never knew when I wanted it
  
• I go through spurts of travel and I ALWAYS have the wrong book with me ... my moods swing notoriously when I'm traveling and what I want to read or learn about swings with it
  
• I get tired of lugging power cords everywhere
  

And so my great Kindle DX experiment began, just about Father's Day (my gift to myself for being a single Dad with little or no time for book enjoyment).

First off, it is very, very easy to spend money with it ... albeit there is a discount on most books that you purchase for the Kindle. Oddly, this doesn't hold true for most Computer Books, they seem to be full price or near full price across the board. I've already purchased and read several books, including the amazing Darwin's Radio and Darwin's Children books from Greg Bear (conveniently in one "book" for the Kindle) and am now working on The Road and Anathem from Neil Stephenson. All bought sitting, literally, from the comfort of my bed or while camping. I have also downloaded quite a few work PDFs and presentations to it and receive The Denver Post electronically each day, automatically delivered.

Here's a hint on Anathem, I bet my back doesn't hurt from lugging it around and yours does ... and guess what, I have a built-in dictionary so I can just move the cursor to a word and the definition appears on the bottom of the screen.

The downside of newspapers is that they are difficult to read and don't include the two best parts ... the comics and the ads. As such, I still recieve a Sunday Denver Post ... but my recycling bin has definitely went down in size.

When folks ask me about it, I put my Kindle experience rather succinctly:

• Book Reading (fiction / non-fiction): A+ (as easy on the eyes as paper, easy to navigate, etc...)
  
• Reference Literature that you Skip Around in: C (VERY difficult to skip around a book and "leaf through" the pages
  
• Newspapers: B-
  
• Ease of Use: B+
  
• Ease of Shopping: B (I usually shop on a web browser rather than the built in store but if you know what you are buying, the built in store on the Kindle is easy enough)
  
• Cost: C- (this has GOT to come down and I completely object to paying for blog subscriptions ... sorry Amazon and Slashdot, ain't gonna pad your profit margins)
  
• Ease on the Eyes: A+ (as good as paper and my eyes don't get all wobbly like on a computer...I was literally reading under the covers so-as to not annoy my son at Scout Camp...and it's not backlit so I had my flashlight with me, it was just like when I was a kid!)
  
• Portability: A+ (I have cut down the size of my backpack by half ... I still have to carry my Mac)
  

The power lasts FOREVER on the thing. Even with 3G on I plug it in at most once a week.

Now here is the BIGGEST complaint I have. Flying to San Francisco they SPECIFICALLY point out that your Kindle must be turned off. People, it barely uses power. Might I suggest you just ask folks to turn off the 3G on the Kindle instead?

I have even put all of our docs for the Sun Storage 7000 Appliance onto it for quick access at customer black or gray sites. Very handy to have along.

Now here is a tip for those of us who carry these around, you will have people tell you straight out that you are a heretic for abandoning "the feel of paper" and contributing to the "demise of the book" (Auntie, if you are reading this...this is for you . It is SIMPLY not true. In fact, there are a few things that actually seem to work for the author:

• There is no after-market for a book, this is frustrating to me but as long as most books are discounted I am willing to deal with it for now. But consider, I can't resell a book that is on my e-reader and I understand that this is a slippery slope to create.
  
• Personally, books, PDFs, and stuff like that contribute to about 30% of the clutter in my house and frankly, I'm not one to enjoy having bookshelves of the things I've read contributing to the dust in my environment...they are pretty much g-o-n-e
  
• I've already saved multiple trees through e-news delivery and Anathem
  

But in the end, it's not for everyone. But after the end, the demise of the paper novel is nigh my friends, it may not be my generation but I guarantee that my son will be 50% e-books at least in his High School days, by college 75% and his children will not buy paper books, end of story.

I notice this post is exceeding the expected length, drop me a line if there is anything you want to know about!

Heroes Included at the OpenSolaris Storage Summit!!!!!!

Sunday Feb 08, 2009

I was just browsing the agenda for the OpenSolaris Storage Summit that is being held on February 23rd 2009 in San Francisco (city with that crooked street ya know).

It turns out one of the big talks is going to be from Don McAskill from SmugMug ... and I kid you not and I'm not doing any schmooozing here, but it is my favorite favorite photo sharing site so I've already signed up. Also on the list are Mike Shapiro or Randy Bias from GoGrid. What does Storage and OpenSolaris have to do with the cloud????

It turns out, everything in the cloud has to end up SOMEWHERE and if its not using the cloud itself as a storage utility (via the likes of Amazon S3, GoGrid and others), the cloud applications themselves use a boatload of storage. The Sun Storage 7000 family and the Sun Open Storage J4000 family coupled with OpenSolaris are definitely the way to go for infrastructure.

Doh, I'm totally geeked and off to the travel site to figure out how to book my trip and get my kiddo taken care of for the day (though as you can see...he loves San Francisco).

IT'S THE little THINGS

Thursday Feb 05, 2009

In the storage industry (and the computer industry in general) we have this term known as a "dissatisfier". A dissatisfier is basically a feature or a product that people have to use that no matter how well you do it, you will almost never receive a "win" (or even an acknowledgment of how well you implemented the feature) since the customer isn't really buying the product for that feature. On the other hand, if you implement the feature poorly, you can lose deals and cause a great deal of consternation on customer sites.

Solaris 10 Update 6 (as well as the latest OpenSolaris release) fixes a huge dissatisfier for me. After choosing what devices to install the system on, in previous versions of Solaris, you were met with a prompt asking if you wanted to partition the storage by default or customize the directory allocations. At this point you would go in and choose how much storage goes into the root (/), how much goes into home directories (/export/home), how much is swap and you could add your own partitions. I know I'm bad, but since I'm an engineer I never use /export/home on my system so I have to jigger the directories and fix how much storage is allocated.

Along comes ZFS Boot. Now, there are a LOT of reasons to love ZFS boot (we use it on the Sun Storage 7000 and on most of our developer systems). On previous projects, we noticed how quick and easy it was to do things like LiveUpgrade.

Now, remember that ZFS is natively thin provisioned...meaning, you can create file systems that all sit over the same pool of storage (multiple drives, devices, enclosures) and each file system vies for the storage (you can add quotas and reserve space, don't worry).

Today I installed Solaris 10 U6 and blew through the installation process without EVER having to finagle a screen saying how much storage should be allocated to each directory. YAAAAYYYYYYYYYYYYY

Basically, you choose to install with ZFS, select two disks (so the install can be mirrored and one device can fail) and off the install goes happily ever after.

Ahhhhh, its the little things ....

BitTorrent on Solaris (Nevada, Solaris 10 Update 5)

Thursday Jul 17, 2008

After years of trying to squirrel away a few minutes to install my only BitTorrent swarm in our lab, I've finally done it! I thought I'd put the instructions out here on the web since it seems that explicit Solaris-based instructions are relatively sparse (they are sparse because the version I'm using is relatively easy to use and not relegated to a single platform...thus the lack of Solaris-specifics).

I started with a clean build of Solaris 10 Update 5 on one machine, and Build 94 of Nevada on another set of 4 machines. These are the Sun Fire x4150s I've discussed in other posts.

Retrieving all of the packages I needed was straightforward with Blastwave.org (an open source software repository for Solaris). In fact, with just a few commands I had a complete BitTorrent stack installed that runs with Python (don't have Python on the system...Blastwave takes care of that too).

Install BitTorrent
First, read the HOWTO on using Blastwave. I only made it through a few steps of the HOWTO before I was up and running, so don't sweat it if it seems long and you are attention-challenged (like me without a special Nespresso Level 8 or above espresso). Of course, I'm fumbling around more than I should so be sure to read the whole thing when you do have time, it contains information on directories and where stuff gets put (at least I think it does...). Installing what you need to use Blastwave takes only a few minutes. Once you have Blastwave ready and running, retrieve BitTorrent and all its dependencies (primarily Python):

/opt/csw/bin/pkg-get -i bittorrent

That's it, BitTorrent is now installed in /opt/csw/share/BitTorrent

Remember the BitTorrent Components
Here is the 2 minute review of important BitTorrent Network Components and terminology that follows the general deployment diagram here (this is the swarm I've installed in my lab so I'll refer to it throughout):

• Torrent - A "file" that is prepared to be distributed by BitTorrent, torrent may also refer to the metadata file prepared from the contents of the file you will be distributing
  
• Tracker - A "tracker" helps refer downloaders (peers) to each other, sort of an organizer of nodes
  
• Swarm - A group of peers that are delivering content to a requester (the downloader itself often becomes a part of the swarm for another requester very early in the download process)
  
• Seed - A full copy of a file that clients can obtain. This seems like a heavy load on a single server, but very early on clients will lessen their load on the Seed systems and depend on other peer clients for chunks of the torrent.
  

In the scenario below, the Tracker and the Seed for my torrent (OpenSolaris 08.05 ISO) are one and the same. The Seed should typically be split from the Tracker though...the Tracker is a bottleneck and you don't want it to be pounded on by Peers.

Start a Tracker
We will start the tracker first. A tracker can be started as a background task. It will sit on a port and listen for requests and write output to a file.

./bttrack.py --port 6969 --dfile dstate --logfile tracker.out

Your tracker is now up and running and ready to organize clients.

Create a Torrent
A "torrent" metadata file contains all of the information about a particular file that can be retrieved through BitTorrent. The torrent is associated with a tracker and when complete will contain a variety of information to ensure the integrity of the final torrent when a client receives it.

The README.txt with the BitTorrent download didn't have proper instructions for building a torrent (seemed a little out of date), so be careful and don't get frustrated. Use the btmaketorrent.py command to build the torrent file. I've chosen to create a torrent from the OpenSolaris 08.05 distribution (my favorite distribution of all, of course...):

./btmaketorrent.py --comment "OpenSolaris 08.05 from Pauls Swarm" http://x4xxx-01.sun.com:6969/announce os200805.iso

Notice the use of the tracker that I set up. You can show the meta information that was generated by using the btshowmetainfo.py command:

# ./btshowmetainfo.py os200805.iso.torrent
btshowmetainfo 4.0.4 - decode BitTorrent metainfo files

metainfo file.: os200805.iso.torrent
info hash.....: fdf239d2524e44432892d01ab354e20a8b77b7e6
file name.....: os200805.iso
file size.....: 719087616 (2743 * 262144 + 26624)
announce url..: http://x4xxx-01.sun.com:6969/announce


Setting up a Seed
You need a web server available from the system that is going to be a "Seed". Nevada has Apache 2.2 (at least in my build) installed by default, you simply have to turn it on (mileage may vary). You can do this by typing:

# svcadm enable apache22
# svcs | grep -i apache2
online 20:41:36 svc:/network/http:apache22

A better instruction set can be found on BigAdmin.

With a little nosing around the system, I found my httpd.conf file at /etc/apache2/2.2. Look in the file for the DocumentRoot, something like this works:

# cat httpd.conf | grep DocumentRoot
# DocumentRoot: The directory out of which you will serve your
DocumentRoot "/var/apache2/2.2/htdocs"


We will place our .torrent file (created earlier) in the DocumentRoot.

Most web servers you also have to associate the mime type "application/x-bittorrent" with the file extension "torrent", this was already done for the pre-installed Apache 2.2 on Nevada.

Finally, put the complete file on a server (the same one would work like I am doing, but a separate server is recommended). Run the BitTorrent download, saving the file into the location that the file already exists. BitTorrent is smart enough to see all of the chunks are there, but it spends time verifying the are correct. When complete, the computer you run the download is a Seed. This command works for the OpenSolaris 08.05 torrent I created earlier:

# ./btdownloadheadless.py --url http://x4xxx-01.sun.com/os200805.iso.torrent --save_as os200805.iso

This takes a while as it runs through hash checks on the existing file.

The torrent is now ready as a Seed for peer to peer access. You can add as many Seeds as you want depending on how popular you think your file is going to be.

Use another Client to Download a File
I can now go to x4xx-02 and start the headless download, just as the previous one ran.

# ./btdownloadheadless.py --url http://x4xxx-01.sun.com/os200805.iso.torrent --save_as os200805.iso

Remember the torrent itself points back to the tracker. There should be some brief activity by the tracker during the download to see if other peers in the swarm can help.

During the download, you will become a part of the swarm from which other clients can download chunks of OpenSolaris (you have to give it, if you want to get it - The Suburbs).

As the file starts to download, you will see the transfer rate start to go up. With a single Seed, I started slowly ramping up. BitTorrent is careful to balance requests. Remember, I now have a Seed and a Peer with chunks of the file. With a single download in progress, I reached about 20 KB/sec (the upload from the Seed and the download from the client)...there is some obvious throttling going on somewhere.

What if I start a download of the torrent on x4xxx-03? The Seed remained uploading around 20 KB/sec. After an initial hit on x4xxx-02 down to about 14 KB/sec, I quickly moved back to over 20 KB/sec, while x4xxx-03 was peaking near 30 KB/sec. As more chunks moved onto x4xxx-03, x4xxx-02 also sped up, since it could grab chunks from two peers. The Seed remained constant at a 20 KB/sec upload, but x4xx-02 was now also uploading at 20 KB/sec and x4xxx-03 was able to have the aggregate bandwidth.

But wait, I have two more clients sitting idle, I started up x4xxx-04 and x4xxx-05 with the download. Again, an initial hit from x4xxx-02 occurred, as it was heavily relying on the original Seed and clients were also relying on it. Within about a minute, the original Seed was still uploading at 20 KB/sec, but -02, -03 and -04 were also uploading at 20 KB/sec. All clients were now downloading at 20 KB/sec with the original seed still uploading at a constant 20 KB/sec...peer to peer amortization of upload bandwidth at its finest.

As the download moved along, more chunks flooded onto the remaining peers and the sharing became much more efficient. Each of the clients regularly found their way over 25 KB/sec and often into the 30 KB/sec range.

I decided that the graphical output wasn't very fun, so I added my home system that I VPN with into the party using Azureus. With Azureus, I'm able to get graphical displays of what's going on. Here is a picture of my swarm:

Note that the center circle is my client, the fully blue circle is the seed, and you can see the other peers don't yet have a lot of chunks of the file.

Here is another interesting view from Azureus:

Of particular note is the Swarm Speed, we have hit about 100 KB/sec with our 5 peers and an obvious upload throttling on each client at around 20 KB/sec. Well, I'll obviously be looking into that...but I think I've run out of words for the evening and I've more than achieved my purpose tonight. Enjoy BitTorrent on Solaris!

Now, I just have to figure out what other kinds of fun I can have with my swarm

Fun with the FMA and SNMP

Thursday Jun 26, 2008

I've been working back in the Auto Service Request space for the past few weeks and doing some education and stuff as we build and test this product or that, it seemed like a great opportunity to put out a quick and dirty "SNMP with the FMA" blog post. There are some excellent previous materials out there that are extremely applicable and you could probably get by with, so first I will point those out:

• A Louder Voice for the Fault Manager - A blog post by Keith Wesolowski that is definitely the start and the finish of of SNMP with FMA
  
• Solaris 10 Predictive Self-Healing and Solaris Diagnostics
  
• The FMA MIB itself
  

With those articles and a little help from your nearby SNMP guru, you are pretty much good to go. I've extended the information in those papers a tiny, tiny bit with how to link the information back to Sun's message repository. The great thing about the FMA is that we have "special sauce" around that can create a full-service cycle (including a Remote Shell application that service can use with your permission and oversight to further diagnose problems from a remote location as well as tools to deliver faults back to Sun Microsystems, including our own Common Array Manager).

Introduction

This guides a user through a 2 system setup with

• system1 - enabled to send traps when a fault is detected by the FMA
  
• system 2 - enabled to receive traps from system1
  

Throughout this write-up, a roughly equivalent FMA command is given that can be run on the originating host so you can follow what the SNMP trap is delivering. To me, the FMA command on the originating host is definitely the preference though, since it usually provides additional information and formatting that may not be available in the SNMP trap or walk.

Setup of Trap Sender

system1 must be setup to deliver traps when FMA events occur.

To do this, follow the instructions provided at A Louder Voice for the Fault Manager, summarized here:

• Create the file /etc/sma/snmp/snmp.conf (if it doesn't exist) and add the line "mibs +ALL"
  
• Add the line to /etc/sma/snmp/snmpd.conf: trap2sink system2
  
• Add the additional line to support deeper information query: dlmod sunFM /usr/lib/fm/amd64/libfmd_snmp.so.1
  

Assuming you have made no other changes prior to this, your system should be ready to go. You do have to restart the SNMP service at this point (I always reboot...its a Windows habit).

Setup of Trap Receiver

system2 must be setup to receive traps, this is simple for demo purposes:

• Run /usr/sfw/sbin/snmptrapd -P
  
• Watch for traps
  

Receiving FMA traps

When system2 receives FMA traps from system1, they will look like this (formatting appropriately rearranged)


DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1190862249) 137 days, 19:57:02.49
SNMPv2-MIB::snmpTrapOID.0 = OID: SUN-FM-MIB::sunFmProblemTrap
SUN-FM-MIB::sunFmProblemUUID."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"
SUN-FM-MIB::sunFmProblemCode."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: ZFS-8000-CS
SUN-FM-MIB::sunFmProblemURL."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: http://sun.com/msg/ZFS-8000-CS


Check out that URL in the sunFmProblemURL, http://sun.com/msg/ZFS-8000-CS. You can actually go there and get the System Administrator Actions that should be taken as well as an extended view of the problem (not contextualized to your view, but this is the general fault information that you would see if you were on the system itself).

This trap is roughly equivalent to the information you would receive from running the
basic fmdump command on the system with the fault. You could also run the "fmdump -v -u 33570acb-e108-4ca8-8276-c67aeecf2043" command on the trap originator to get a bit more information:

bash-3.00# fmdump -v -u 33570acb-e108-4ca8-8276-c67aeecf2043
TIME UUID SUNW-MSG-ID
Jun 26 10:57:45.3602 33570acb-e108-4ca8-8276-c67aeecf2043 ZFS-8000-CS
100% fault.fs.zfs.pool

Problem in: zfs://pool=data
Affects: zfs://pool=data
FRU: -
Location: -


Navigating Problems on the Remote System

Let's start at the top now. We can quickly navigate all known, unresolved problems on the remote system by walking the sunFmProblemTable:

system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmProblemTable

This results in a dump of all problems with the UUIDs that can be used for deeper queries. The following printout is only a single problem on the system, additional problems are listed in blocks by the attribute (so all sunFmProblemUUIDs are lumped together followed by all ProblemCodes, and so on).

SUN-FM-MIB::sunFmProblemUUID."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: "0f3dcdf3-f85b-c091-8f1f-ce2164976cda"
SUN-FM-MIB::sunFmProblemCode."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: ZFS-8000-CS
SUN-FM-MIB::sunFmProblemURL."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: http://sun.com/msg/ZFS-8000-CS
SUN-FM-MIB::sunFmProblemDiagEngine."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: fmd:///module/fmd
SUN-FM-MIB::sunFmProblemDiagTime."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: 2008-6-18,10:7:4.0,-6:0
SUN-FM-MIB::sunFmProblemSuspectCount."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = Gauge32: 1


This is roughly equivalent to the fmdump command on a system, though with the basic fmdump only the UUID, Code and a MSG-ID are given.

Based on this information, we can look at the ZFS-8000-CS message at sun.com and determine what our next steps should be. It indicates that using the zpool status -x on the system with the fault would be useful. Going to the originating system and running it returns:


bash-3.00# zpool status -x
pool: data
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
data UNAVAIL 0 0 0 insufficient replicas
mirror UNAVAIL 0 0 0 insufficient replicas
c2t0d0 UNAVAIL 0 0 0 cannot open
c3t1d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t2d0 UNAVAIL 0 0 0 cannot open
c3t3d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t6d0 UNAVAIL 0 0 0 cannot open
c3t7d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t8d0 UNAVAIL 0 0 0 cannot open
c3t9d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t10d0 UNAVAIL 0 0 0 cannot open
c3t11d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t12d0 UNAVAIL 0 0 0 cannot open
c3t13d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t14d0 UNAVAIL 0 0 0 cannot open
c3t15d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t16d0 UNAVAIL 0 0 0 cannot open
c3t17d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t18d0 UNAVAIL 0 0 0 cannot open
c3t19d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t20d0 UNAVAIL 0 0 0 cannot open
c3t21d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t22d0 UNAVAIL 0 0 0 cannot open
c3t23d0 UNAVAIL 0 0 0 cannot open
bash-3.00#


For some history on this particular problem, we disconnected a JBOD that had the "data" pool built, so none of the devices are available...ouch.

You can look more deeply at the events that resulted in the problem by walking the sunFmFaultEventTable:

system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmFaultEventTable
SUN-FM-MIB::sunFmFaultEventProblemUUID."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: "0f3dcdf3-f85b-c091-8f1f-ce2164976cda"
SUN-FM-MIB::sunFmFaultEventProblemUUID."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"
SUN-FM-MIB::sunFmFaultEventProblemUUID."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: "3600a05e-acc1-cae2-c185-f50852156777"
SUN-FM-MIB::sunFmFaultEventProblemUUID."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: "97bfeb63-7b02-c2b6-c51f-c451a9f760c5"
SUN-FM-MIB::sunFmFaultEventClass."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventCertainty."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."33570acb-e108-4ca8-8276-c67aeecf2043".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."3600a05e-acc1-cae2-c185-f50852156777".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventASRU."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: zfs://pool=1ca09fa50e7ca8c7
SUN-FM-MIB::sunFmFaultEventASRU."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventASRU."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: zfs://pool=6f658a6c4b99b18b
SUN-FM-MIB::sunFmFaultEventASRU."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventFRU."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventResource."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: zfs://pool=1ca09fa50e7ca8c7
SUN-FM-MIB::sunFmFaultEventResource."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventResource."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: zfs://pool=6f658a6c4b99b18b
SUN-FM-MIB::sunFmFaultEventResource."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: zfs://pool=data


This is roughly equivalent to the fmdump -v command. The fmdump -V command cannot be duplicated over SNMP, though it can be useful to run on the host side. fmdump -V can provide product, chassis and server IDs as well as a more complete list of faults and ereports that resulted in the diagnosis.

You could also view the fault management information by resource rather than by fault or event over SNMP

system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmResourceTable
SUN-FM-MIB::sunFmResourceFMRI.1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmResourceStatus.1 = INTEGER: faulted(5)
SUN-FM-MIB::sunFmResourceDiagnosisUUID.1 = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"


This is similar to the "fmadm faulty" command that can be run from a system. The faulty parameter results in the some additional information and text, though that text can also be retrieved at the event URL that was earlier identified.

While this is the "SPAM" approach to inquiring about a system, you could also walk each problem as they come in.

As was mentioned previously, there isn't a "ton" of information available within the SNMP MIB itself. The result of receiving a particular trap is often to do additional diagnosis on the system with the fault. So some ssh work may be necessary.

Additional things to do with FMA

You can dump the topology of a system!

Go to /usr/lib/fm/fmd and type "fmtopo", you should get a complete topology as it is recognized by the fault management infrastructure.


bash-3.00# ./fmtopo
TIME UUID
Jun 26 13:20:16 ddba8792-0166-6fb4-a81b-db3de9622649

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=1

hc://:product-id=Sun-Fire-X4150:chassis-id=0811QAR189:server-id=server1/motherboard=0/chip=0/cpu=2

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=3

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0/dram-channel=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0/rank=0

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0/rank=1

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0/dram-channel=1

hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=555555555:part=777777777 :revision/motherboard=0/memory-controller=0/dram-channel=1/dimm=0

... (and on and on)


Artificial Fault Injection

There isn't a lot of documentation I've found for fminject, but it looks like it can be quite useful. It will, at its easiest, allow you to replay a set of ereports and faults that previously occurred in a system. For example, we can replay the error log that resulted in the FMA trap for the ZFS pool problem that the trap was sent out for. In one window on the system that sent the trap, start up the simulator

cd /usr/lib/fm/fmd
./fmsim -i


In another window, replay an error log

cd /usr/lib/fm/fmd
./fminject -v /var/fm/fmd/errlog


You will see the SNMP Trap re-appear in the trap receiver. Note there is no good way to determine that this is a replayed trap in the trap receiver's window that I've determined, so use this with caution.

Summary

While there is enough information to completely diagnose a problem from an SNMP trap, the traps are a great opportunity for integration with higher level management tools. More often than not, additional commands and information are required to be run on the system with the fault (be it logs or output of various commands, especially in the ZFS space), but the messages themselves typically have the next steps for diagnosis.

When the clouds disappear...not always a sunny day

Friday Jun 20, 2008

I build, maintain and pay for hosting for my friend's charity web site, Play for a Heart. For the past several years I've had a host provider where I would deploy the very simple web site I constructed for her. The host had become my "cloud" in the stormy skies of Cloud Computing.

This cloud kept drifting along, the price was right, it was somewhat reliable, it was ... easy.

I went to the web site on a Saturday morning, it was gone. Somewhere over the course of the last few months the company had turned into a shell of the host provider it formerly was. There was no tech response over the weekend and the forums were more or less ghost towns. I made a post and the only responses were "Leave this host as fast as you can". The outage continued on through the weekend and extended to a large percentage of the host's own sales and marketing sites.

Finally, late Sunday I couldn't take it anymore, I pulled the plug and moved to GoDaddy. The move had nothing to do with Danica Patrick. Here was my very simple logic:

• Java Support / PHP Support / MySQL Support
  
• Able to host multiple domains with a single hosting account
  
• Resilient enough to support the onslaught after a Super Bowl commercial
  
• Good recommendations and community
  
• Great price
  

And I switched...

Lucky for me I made my friend's site as basic as I could.

• Very simple HTML
  
• Slideshow objects embedded from another Cloud Application (SmugMug so that the links moved right over (and I only put these on after she had enough sponsors that I couldn't figure out how to keep the sponsor page clean)
  
• No applications embedded directly from my low-budget hosting solution
  
• Domain Names purchased from Yahoo and GoDaddy (this became a HUGE win as the host records would have been locked had I purchased the domain names with this smaller company)
  

My web site was moved to GoDaddy and back up and running in, literally, 1.5 hours and my exit was complete from my previous host.

Still, the entire experience left me very shaken over Cloud Computing. I've come up with the following set of thoughts when it comes to attaching my digital life to the clouds:

• Many companies literally own the information you create or have very liberal rights to that information (Always read the License Agreements)
  
• Many companies have no exit strategy for your information (once your data is captured or created in the online application it cannot be extracted...this is especially the case in social networking infrastructure)
  
• Because of the nature of the cloud, you have no guarantees that many companies that host the applications you depend on are even viable (check the business model and financials if you are tying your life to the cloud)
  
• Several companies entire business models are centered on analyzing the information you give to them and monetizing it (in many cases "personalization" in your eyes is really "targeting and demographics" in the company's eyes)
  
• Standard Platforms, Standard APIs, Standard Information Models are at the heart of a successful cloud (not fog), this allows you a better chance at having tools to import / export information and interact with that information...and in the case of building an application, it is critical that these standards are the core of the application so that you can get out of your cloud as quickly as you got into it and at the first sign of turbulence
  
• In lieu of standards (even defacto), Open APIs, Open Source, Open Architecture, and a good open license can be a huge help...especially with a robust developer community. This assures infrastructure can live beyond the life of any individual cloud if it should disappear.
  
• Buyer beware
  

Well...I'm a huge, huge advocate of Cloud Computing. Our own network.com site, Amazon EC3 and S3, SalesForce.com, they are all leading the wave...and in 10 years it will just be the way it is done. But until then, there will certainly be some growing pains, probably more in the small-company space than in the enterprise space...but they will be there.

Standard's groups, defacto standards, open source cloud infrastructure, entry and exit for information, information licensing, information security and transparent motivation for collection of your information will go a long way to determining whether there is longevity in the sunny skies for cloud computing.

Preservation and Archive Special Interest Group (PASIG)

Thursday May 29, 2008

I am in San Francisco at Sun's Preservation and Archive Special Interest Group (PASIG) Face to Face meeting. Preservation and Archiving is one of the most fascinating and complex problems to face our digital world, in my humble opinion. I've chatted about this before in the blog so I won't belabor the point that taking a chisel to the walls of this beautiful location and etching in the contents of your e-book would have a longer life span than most of our media types in use today.

Still, chiseling has some pretty serious downsides

• It doesn't scale well (oh, and you have to scale to get up the wall)
  
• You have to travel to this site to view the contents (while that might be pleasant, it can be quite inefficient)
  
• Its difficult to create a geographically remote backup site
  
• It seriously blemishes a beautiful location
  

It turns out, there is a community of people that think about this problem as their day job and PASIG is a gathering point for that community. The community is made up of the "new" digital librarians, software that provides the foundation for digital preservation and archives, storage vendors that can provide "temporal" storage locations for the digital archive, and system integrators that can customize the entire stack and tailor it for a particular domain (astronomy, human sciences, etc...).

The community is global, but their is an obvious disparity in how this type of work is funded across countries and boundaries. Here in the U.S., the leaders of preservation and archiving efforts appears to be the education community, with a variety of colleges (Penn State, University of Pittsburgh (Pennsylvania), etc...) and individual government agencies (Library of Congress, National Archives, etc...). Abroad, it seems the repositories are often nationally funded.

While various technologies were interesting, it was also interesting to hear the oft recurring theme of "sustainability"...how can we "sustain" the rate of ingest and cost over time of these types of solutions? Consider the cost of the initial storage of 1 GB of data today (just the storage medium), its between $0.25 and $1.00 depending on your storage medium. Now add in the cost of the various transitions that data makes over 20,000 years (life of a cave painting) the system administration, the building of new equipment, the power involved, etc... At this stage of human evolution, it is difficult to even predict what type of media we will store data on in 20 or 30 years. How can one predict the cost of sustaining these archives, when we won't know what the archive will look like, let alone the environment around it, in 20 years, 100 years or even 1,000 years.

Which brings up another interesting thread...whereas a cave painting is required to stay on the rock it originated on, the life of preserved digital data (and archived) must be assumed to migrate over the data generations. Consider each generation of media about 4 years (3-5 is common), tape can technically survive multiple generations and one can build systems (like the Sun Storage 5800 (Honeycomb) platform) that can theoretically store data for 100,000s years. But the reality is that the lifetime of a technology and a media is that you migrate approximately every 3-5 years.

This is a conceptually different model then many people think of when they think of an "archive" or a "storage vault". Rather than think of a monolithic system that weathers the years, its important to think of data as migratory, and evolutionary (much like humans themselves). To survive 30,000 years, our digital data originates...often in the physical world...but increasingly our data originates in the digital world itself...and migrates to a Tier 1 storage device. Regardless of whether it remains there, over the next 20 years, that data will move across 4 or more devices (not counting replicants or backups) if it stays "online".

So what are the three most important attributes of an archive and/or a preservation system (IMHO):

• The information is safe, stable, and can be verified to be "original" while it resides temporarily within the system
  
• The information can be moved off the system when the time is right and can still be certified as "original" data rather than a derivation of the data
  
• Information can be "moved into" a new preservation / archive system.
  

In the end, long-term data is migratory...and our systems and software that support this field must start thinking of data that way. This type of thinking can also be applied backwards into more traditional storage paradigms, how useful would it be for a photo repository to be able to migrate to bigger and bigger system with higher capacity and lower power consumption? At some point, there is a trade-off that will trigger that company to move, and facilitating that move is in the best interest of all companies.

What else came up at the conference? Tons of things, have we thought enough about:

• Access Rights (and access across time)
  
• Roles and Responsibilities in a digital archiving and library maintenance system (the new librarians)
  
• Geo-political boundaries as repositories move into the "cloud"
  

At this very minute, Sam Gustman is discussing The Shoah Foundation. The mission of The Shoah Foundation is To overcome prejudice, intolerance and bigotry - and the suffering they cause - through the educational use of the Institute's visual history testimonies. The documentation of The Holocaust contains over 120,000 hours of online video testimony. The foundation indexes minute by minute of this testimony and makes it accessible for people to teach and understand the impacts of the holocaust. The foundation moves beyond The Holocaust into testimonies of other events in human history that must be recorded and understood. As George Santayana said: Those who do not learn from history are doomed to repeat it.

Sam's presentation isn't online, I'm trying to get Art to post it. This work alone shows that digital archiving and preservation is not just an interesting segment of compute and storage...its a moral imperative.

I found this link to a video that discusses Fedora Commons, one of the open source repository software initiatives for creating and managing digital libraries and archives. The video discusses some of the solutions that are being built with the Fedora Commons software solution:

What a great opportunity the conference has been to sit and think about the big picture (and archiving and preservation truly are the big picture).

Sun Open Storage and the Sun Fire x4240

Thursday May 15, 2008

We announced a whole set of new servers this week, including one of my favorites, the Sun Fire X4240 Server. I only like to talk about things I've touched and played with so I figured I'd show you a little about how I'm using my x4240s.

I'm a storage guy, so what I immediately like about this particular system (and the Sun Fire x4150s I talked about before, as well as the related 8 and 16 disk offerings with SPARC chips), is the disk density I get along with the raw horsepower sitting right next to the disk drives.

The Sun Fire x4240s I'm using right now have 2 x Dual-Core AMD Opteron 2222s that run at 3.0 GHz, but you can have a couple of Quad-Cores in this thing as well! As for storage, I have a set with 146 GB, 10k RPM SAS Drives that are pumping about 66 MB/sec through a single NIC port with our Open Storage CIFS Server that integrates nicely with ZFS.

Watch me share 15 drives through CIFS so those poor Windows folks can attach to my box for their storage (and get the benefits of ZFS snapshot and capacity):


$ zpool create sharepool raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 raidz2 c1t6d0 c1t7d0 c1t8d0 c1t9d0 c1t10d0 raidz2 c1t11d0 c1t12d0 c1t13d0 c1t14d0 c1t15d0
$ zfs create sharepool/share1
$ zfs set sharesmb=on sharepool/share1
$ zfs set sharesmb=name=share1 sharepool/share1


(You also have to configure the PAM module appropriately, but this is a one time thing, try this:

1. Add the below line to the end of /etc/pam.conf:

other password required pam_smb_passwd.so.1 nowarn

2. Set/Change the Passwords for any Solaris User That Will be Used to Authenticate when Connecting to a CIFS share, like this:

$ passwd pmonday
New Password:
Re-enter new Password:
passwd: password successfully changed for pmonday


With the SMB PAM module installed, passwords are generated that can be used by Windows as well as Solaris (you have to do this). There are, of course, differences from your normal Samba installation but with everything, figure out the tool that is best for you and go with it One thing you notice with Sun's CIFS Server is there is no anonymous access to shares (this can be a good thing)!

Sun's CIFS Server and the Sun Fire X4240 are just one example of how the Open Storage initiative pays off. You can turn this capacity / performance rich server into an iSCSI target, NFS target, or even keep the storage to yourself and load applications directly on the box that can make use of the fast SAS spindles. Don't want to use Solaris or OpenSolaris...heck, load Red Hat or Windows on these things, we do it all the time...in fact, we'll even pre-load a Linux-variant for you (check the options page).

Cool stuff, nice job System's Group! Oh yeah, and the iLOM in the x4240 works GREAT (better than the x4150 even).

Paul [Hearts] eLOM

Friday Apr 25, 2008

For the past week or so I've been on one of those "get this done" projects from my boss. Its tough to "get this done" when you are commuting though (or when the lab is the same temperature and wind speed as the top of Loveland Pass). My project involves 5 Sun Fire x4150s (and will involve another set of servers next week).

I did have to go into the office for a few days to pull memory in and out of them, do some DVD swapping on the machines, check cables, and (of course) get the 8 x 73GB 15K SAS drives swapped in (all of my machines need the same drives and speed).

During the setup, Terry Hill allocated IP addresses for the service processor eLOM (embedded Lights Out Manager) and configured the IP addresses within the BIOS. Keep in mind when you look at the back of a Sun system with eLOM, you see the standard ethernet ports (typically 4 on this size box), but you also see an additional port that's offset from the others and labeled "Service Processor". It is next to the Serial Console port. You can see it here, on the left side of the back of the system.

(The x4150 eLOM documentation has more details about configuring eLOM).

Once the network is configured, you type in the web address of your x4150 service processor (http://myx4150.xxx.com) and up pops the eLOM Web User Interface. You can do this from anywhere (even home

Now that I'm in the eLOM, I can do all sorts of things. I primarily use the Web interface. With it I can do all of the power resets I need to do (including having it come straight up to BIOS). I can also launch a console through the magic of Java. When you press the button to get a console redirection, a tiny Java Web Start application comes down to your machine and connects back to the server. With it I can do all sorts of cool things like handle BIOS, take console options on my Operating Systems (I use Solaris, OpenSolaris, RHEL and Windows on my machines), and so on... I am also able to monitor event logs and ensure the components are operating as I expect (heck, I can even flash LEDs to say Hi to all of those frozen lab folks who haven't set up their eLOM).

My biggest complaint with eLOM isn't even with eLOM, its with Mac OS X. I have to tell you, I am not happy with their Java support at all. The eLOM console redirection doesn't work (strange Apple-only Java problem). I went to java.sun.com to get a Mac OSX updated JVM...no luck, have to go to Apple. Go to Apple, they are hung up on Java 1.5. I found the Java Platform Standard Edition 6 Press Announcement, its been out since December 2006.

I found a Java Platform Standard Edition 6 Developer Release at Apple but it only runs on Leopard and I haven't upgraded (and I'm not). So, I'm stuck with the Mac...can't use it in our labs to support my systems. I know people love their Macs but some days it reminds me a little too much of OS/2. Great promise with a lack of application support.

I bounced to another platform (with its own Java 1.5 that actually works), the JVMs on my Solaris machines work fine as well as on my Windows laptop and home systems.

In the end...lovin' the eLOM!

ATTACK OF THE PLASTIC GROCERY BAGS

Monday Apr 21, 2008

Have you ever been attacked by a plastic grocery bag on the way to work? One of those ones that float through traffic freaking you out? And then it seems to somehow manage to go under your car and pop out the other side only to terrorize yet another driver! Kai and I did some calculations with the bags.

Each bag is a bit over a 15 inch square.

If I didn't use my cloth bags and reuse the plastic "overspill" bags, I would use about 10 per week (why they pack a single bag of potato chips in another plastic bag all by itself is pretty far beyond my comprehension). So, I work my butt off to save at least 5 plastic bags a week (usually I can save all 10). Its probably reasonable that every family could cut about 5 bags off their plastic bag diet if they haven't already.

Saving my 5 bags each week saves about (5 bags * 52 weeks) 260 bags per year. That's saving about (260 bags per year * 15 inches) 3,900 bag inches long (15 bag inches wide) and that is about (3,900 bag inches / 12 inches per foot) 325 feet. Each year saves about the number of bags to go across an American football field and an end zone.

(Yes, that's the high school rugby team playing on the football field...I know they don't need a bunch of plastic grocery bags flying across the field...and they do, quite often).

Let's say all of the families in Highlands Ranch (about 20,000) stopped using about 5 bags a week too. That would be 325 feet * 20,000 people = 6,500,000 feet (15" wide). That's 1,231 miles of bags (15" wide).

The folks of Highlands Ranch could save a strip of plastic bags that runs from Denver to San Francisco (as the Google Map flies).

View Larger Map

Well, that's a lot of bags. Who knows the countless number of car accidents that would be avoided if we all saved a few plastic bags. Reuse the ones you have and get, recycle them, and in so many cases...don't even use them in the first place.

While thinking about the surface area that these bags cover is convenient and somewhat astounding, also keep in mind

• Energy is used to produce these bags
  
• The bags are comprised of materials like plastics and oils
  
• The bags take up space in landfills (not much...but it is space)
  
• The bags won't decompose for 100s or 1000s of years
  

Next time just think: "Do I really need that bag or can I just carry my sack of potatoes?".

Power Consumption of Hanging Clothes out to Dry

Monday Apr 14, 2008

I had a section to do for Kai to become a Wolf in Cub Scouts...luckily, the pack has decided to go out and do some neighborhood cleanup. Still, I was interested in completing my fact finding mission anyway. My goal was to create a set of information on how "little things" add up to "big things" (remember "think globally, act locally").

There is a lot to choose from in my life, I figured I would take them one by one and see if a little "group think" can help me fix any logic problems. After the Jack Johnson Curious George Soundtrack song, I figured I would categorize into:

• Reduce
  
• Reuse
  
• Recycle
  

And I would take a few simple tasks a child could complete with their parents:

• Encourage your parents to hang out 2 loads of clothes a week
  
• Stop using 5 plastic bags a week
  
• Use refillable water bottles instead of purchasing bottles of water
  
• Walk to school
  

I haven't dug up all of the facts yet, but I'm committing to it here on the blog!

First task: Encourage your parents to hang out 2 loads of clothes a week.

(Yes, that's one of the loads I hung out today)

I have an LG Tromm Dryer (courtesy of a good quarter or two here at Sun Microsystems). It runs at 240 Volts / 30 Amps, ouch. That's about 7200 Watts per hour. A load of laundry takes about an hour to dry. I've committed to hang out 2 loads per week of laundry, 14,400 Watts per week * 52 weeks = 748,800 Watts saved per year.

I was browsing around the web trying to find the conversion and I found the Ask a Scientist Web Site that said 1gm of coal could power a 60w light bulb for 550 seconds (let's do some rounding and just say 9 minutes). Now we have to do some normalization.

A 60 Watt light bulb with 1 gm / 9 Minutes = 6.67 grams / hour.

My two loads of laundry actually use 14,400 Watts per week (that's two hours of power), for 1600 grams of coal per week * 52 weeks = 83,200 grams which, for my fellow U.S. citizens is 182 pounds.

Another site estimated the creation of 1 Kilowatt for each pound of coal (roughly), which would bring us to about 748.8 pounds instead of 182. The first calculation was in an ideal world, the latter appears to not be in an ideal world.

Doesn't seem like very much does it. I'll use the 182 figure to be conservative. Each of those pounds of coal produces about 3.7 pounds of CO2 so on the conservative chart, I am stopping about 673 pounds of CO2 from going into the atmosphere.

That's quite a bit, but is it worth the trouble?

My little suburb has about 80,000 folks in it, let's say there are 20,000 homes and each one of those families could save 136 pounds of CO2 from going into the atmosphere. That would be about 13,460,000 pounds of CO2 that does not enter our atmosphere each year.

Is that interesting yet? Probably not in the context of all of the CO2 created in a year, but I think that is starting to get interesting.

For a child, what is 13,460,000 pounds?

I saw on Yahoo Answers that a Pontiac Vibe is approximately (and conveniently) 2,700 pounds.

(Image borrowed from Edmunds)

My little suburb of Highlands Ranch, if each family would hang 2 loads of wash out per week rather than dry them, would keep about 4,985 Pontiac Vibes from entering our atmosphere and hanging over us each year. Imagine what a country like the U.S. could do by hanging their clothes out to dry?

If you see any MORE logic errors, please let me know and I'll keep trap and fix them.

Note: I was off by a power of 10 (sloppy night time work) and my dryer was 240 volts. Michael Lyle sent the correction and also noted that I should use an ammeter to get an actual measurement. Stay tuned!

National Archives, PASIG, a little Vacation

Thursday Mar 27, 2008

My family and I took a brief vacation this weekend and made our way to Washington D.C. for a little R & R. We enjoyed 2 and a half days of sights, tours, history and we even squeezed in a little time for the pool. For those of you that have been to D.C. (or live there), you know that 2 and a half days only allowed us to scratch the surface of the United States cultural base that is alive as well as preserved in the city (and often within a few blocks of the National Mall).

There are so many thought provoking and emotional moments as you move around that after two and a half days I found myself almost completely wrung out. We saved the Congressional Gardens with the Vietnam Memorial, World War II Memorial, Lincoln Memorial, Korean Memorial, and the others for the last day. The artistry and the thought that went into these memorials is astounding and the emotions that they pull out of you put you into knots.

I won't list everything we did on the whole journey over the weekend. For my youngest son, going up the Washington Memorial (and our need to start standing in line for tickets at 6:30am) will probably be the most impacting moments. For Shaun, hopefully the Vietnam Memorial and the Pederson House. For me, who knows, the Bill of Rights, the Constitution, the Declaration of Independence, the Magna Carta...simply amazing, but overall I can't name a single moment that wasn't worth its weight in gold.

Professionally though, the National Archives had to be one of the most thought provoking of our stops.

Here is this large building, with all of these physical manifestations of our history on display and in vaults around the building. The Constitution, the Declaration of Independence, the Bill of Rights, the Emancipation Proclamation and more than I could ever list here. It was "Magna Carta Days" at the National Archives as well and one of the four remaining copies of the 1297 Magna Carta from King Edward I was on display.

Here are a few thoughts that went through my head:

• With infinite, perfect copies of digital content, what makes a digital entity "unique" and "awe inspiring"?
  
• How is our country going to preserve and revere a digital creation 710 years from now?
  
• How do you know what digital creations are worth preserving since you can hit a button and destroy them so easily?
  

The first of these seems entirely out of place with storage technology, but when you stand in front of the Declaration of Independence it makes you wonder what digital content could actually have this impact on a person and how you would embody that digital content. The record companies are struggling with it as well. In addition to digital download content, the companies are trying out releases on USB thumb drives as well as larger packages and "Deluxe" sets. Books are going to struggle through the same revolution (as magazines already have) of being bundled as bits with little or no "branding" or "artistry" about the packaging. How does one recreate that sense of uniqueness when the content is merely a bunch of bits that gets flattened into 10 songs amongst 8,000 on an iPod? Really, what "value" do those songs and books have anymore when they can be passed around at will and are part of a great "torrent" of traffic into and out of our computers? Something really has to "stick" to remain on "top of our stereo" these days.

And as for the Declaration of Independence...what a clean and simple document. The document itself hung in windows and is incredibly faded and worn down. Only after time passed did our country seek to formally preserve it for posterity. Perhaps we caught it in time to save it from deteriorating any more. But one still has to ask, is it the single original document that retains the significance or is it the content that remains significant. If it is the content, we wouldn't store the original in a huge underground vault and protect it as well as Vice President Cheney, would we?

Having seen the original, I would have to argue that there is something incredibly unique about it, it actually holds more reverence (for lack of a better word) than one of the many copies of it. So how does one reproduce that "reverence" in a digital world?

If that is not enough to think about, we have to think about digital preservation. The Magna Carta of 1297 has withstood time for 710 years and is in wonderful shape. What digital storage technology today do we have that can withstand decay for that length of time (of course, one could argue that some rock etchings have withstood time for 1,000s of years). Let's put this in perspective, today's disk drives and SSDs are generally spec'd for 5 years. If I want to preserve my family's pictures for 710 years, I would have to ensure the data was migrated 142 times. Hmmm, I'm not sure if my kids and their kids and their kids are up for that.

It appears that CDs and DVDs may have a lifespan of around 50-200 years if you preserve them properly. That is getting pretty reasonable...of course, they haven't been around for 50 to 200 years so they are certainly not battle tested like carving on a good rock. The National Institute of Standards and Technology appears to be looking heavily into the longevity of optical recording media. DLT appears to have a shelf-life of around 30 years if preserved properly.

Let's say, hypothetically, that you solve the problem of the storage media (perhaps a self migrating technology in a box that guarantees infinite lifespan and that, itself, produces the new disks and technology to ensure fresh DVDs are always built). Now you have two additional challenges (at least):

• Maintaining the integrity of the data (how do I ensure) that the data that is NOW on the DVD is the original data
  
• Maintaining the ability for outsiders to inspect and recall the stored information
  

The first of these seems obvious, but is actually quite difficult. Checksums can be overcome with time (imagine compute power in 700 years!) and we can't guarantee that the keepers of the information will not have a vested interest in changing the contents of the information. We see governments attempting to re-write history all the time, don't we?

Let's take a simpler example of what happens when "a" byte disappears. Recall Neil Armstrong's famous quote: "One small step for man ... ". Well, after a lot of CPU cycles and speculation and conspiracy theories, it turns out that we now believe that Neil Armstrong said: "One small step for a man...". It is a fundamentally different statement (though there is no less historical impact). This data is only 40 years old, but consider the angst in trying to prove whether or not "a" was a part of the quote. What happens when a government deliberately alters, say, the digital equivalent of an "original" 2nd Bill of Rights written in 2030?

One more thought for the day, since I really do have to work and if you have made it this far, it is my duty to you to free you of my ramblings.

We know for a fact that the English (dare I say...United States dialect) language is evolving. Even after 200 years there are phrases and semantics and constructs in the Declaration of Independence that require quite a bit of research for the common US citizen. Take the following paragraph:

He is at this time transporting large Armies of foreign Mercenaries to compleat the works of death, desolation and tyranny, already begun with circumstances of Cruelty & perfidy scarcely paralleled in the most barbarous ages, and totally unworthy the Head of a civilized nation.

There is the obvious use of the word perfidy, a word that has since all but disappeared from common speech in the United States.

Looking deeper at the paragraph we see evolution in spelling (compleat). There is also a fascinating use of capitalization throughout the Declaration of Independence. The study and usage of capitalization alone could be worth the creation of long research papers.

What does this tell us? The content and meaning of a work lies often with the context and times in which the work was created. How does one retain this context, language, and ability to read the content over 700 years? This is not a small problem at all. There are entire cultures lost or in the process of being lost as the language and the context is lost, consider the United States own Anasazi culture as an example.

A computer dialect (protocol, standard, information model, etc...) are themselves subject to evolution and are even more fragile than spoken language itself. A change in a capitalization in an XML model may break the ability of pre-existing programs for reading and migrating information, resulting in lost information. Once you break a program from 200 years prior, how much expertise will still exist to maintain and fix that program?

Crazy things to think about. Personally, I believe we are in a fragile place in our history where we could lose decades of historical information as we transition between written works and digital works. As part of my night job I'm trying to get more involved in the Sun Preservation and Archiving Special Interest Group (PASIG) to learn more about what our customers are doing in this area. I'm also trying to reorganize my own home "infrastructure" to be more resilient for the long run to ensure that my family's history does not disappear with my computers.

There are significant challenges in the computer industry all over, but preservation of history is one that our children and our children's children will judge us with. USB thumb drives will come and go, but hopefully our generation's digital treasures will not go to the grave with us.

Computational Photography and Storage

Monday Mar 03, 2008

There is a great article on CNet's news.com about computational photography, "Photo industry braces for another revolution". It is basically about Photography 2.0. The first wave of digital photography seeks to reproduce film-based photography as well as it can. Photography 2.0 advances hardware while taking advantage of higher processing power within the camera to take advantage of the new hardware, replace hardware functionality with software functionality or bring image detection and manipulation capabilities that are not possible in the hardware space.

There are a few developments worthy of note, and all of them involve bringing more CPU capabilities into the camera:

• Panoramic photography - I enjoy these types of scenes (one shown below), though I don't think they are the future of photography at all
  
• Depth of field and 3-D photography - There is an excellent example of this in the CNet article. Personally, depth of field is arguably one of the most difficult techniques to master since this is purely 4-dimensional using our current lenses (aperture size decrease increases time of exposure and more depth will be in focus, etc...)
  

There are many other ideas in the article...detecting smiles (an extension of this is closed or open eyes), better light detection, self-correcting for stabilization (this is done with high priced hardware today in Image Stabilized lenses), etc... Clearly a Photography 2.0 revolution is in the works.

Photography 2.0 is really the same trend we see in the storage business...Storage 2.0. There are simple changes in the industry, like the incredible increase in CPU driving software RAID into storage stacks again. A huge benefit with software RAID is the decrease in hardware costs that it drives. This is very similar to the Photography 2.0 concept of moving image stabilization out of the hardware (the lenses) and into the software.

Storage 2.0 also brings us projects like this one: Project Royal Jelly. Project Royal Jelly encompasses two important pieces, one is the implementation of a standard access model to fixed content information, a second is the insertion of execution code between the storage API and the spinning rust. The ability to "extend" a storage appliance (or device) via a standard API will allow us to leverage the proliferation of these inexpensive and high-powered CPUs. A common use-case for an execution environment embedded in a storage device would be an image repository or a video repository. Every image submitted goes through a series of conversions: different image formats, different image sizes (thumbnail, Small, Medium, Large), and often a series of color adjustments. Documents go through similar transformations: a PDF may have different formats created (HTML primarily), the document will be indexed, larger chunks will be extracted into a variety of metadata databases for quick views, etc...

These transformations can arguably be the responsibility of the storage operation, not the application operations, especially when the operations can be considered part of an archiving operation. While indexing and manipulation could be considered a higher tier, storage tiering and taking advantage of storage utilities could also benefit from a standard storage execution platform. Vendors could easily insert logic onto storage platforms to "move" data and evolve a storage platform in place rather than authoring applications that have to operate outside of the storage platform.

Just some Monday morning musings...have a great week.

DVD Player = Trash Bin

Monday Feb 25, 2008

My nearby Hollywood Video shuttered its doors this weekend. My MVP membership was automatically canceled by the company. As far as the personalities go, I enjoyed all of the folks there, it was nice to get real recommendations from real people. As I stood in line with my son to purchase a few stray DVDs I talked to my son that in 3-5 years he would never step into a video store again. I got a few stray looks in line. The more I told him about never being in a video store or a music store again, the more I realized what a foolish purchase those plastic cases and metallic disks in my hands were.

I convinced my son we didn't need the DVDs and I promised he could login to Amazon Unbox on the Tivo and rent a movie when we got home. We set the DVDs down, left the line, and I walked out of a video store with my son for the last time ever.

We went home, rented "Mr Bean's Holiday" at home (no, it wasn't my choice) and I'm boxing the DVD player and putting away whatever movies I do have.

And here is a message for Blu-Ray. I haven't looked at a Blu-Ray player, I'm not looking at a Blu-Ray player and I swear I will never have one in my house, period (unless I purchase a video game system with one but then I won't get movies for it anyway). Revel in your victory oh Blu-Ray backers, but it will be short lived, physical media for content is short-lived. I actually think Microsoft XBox is fine if they can get the video downloads working ASAP on their platform. These are pro-gamers, renting online will not be a problem and I doubt pro-gamers show off their DVD collection.

When I don't have my Tivo with me, I can download to my Media Center PC and plug it into the TV through the S-Video port, works great, portable media, very nice.

One less black box in my house is a good thing. Thanks for your time of service DVD player, I'll show you to the door.

Why one bit matters.

Friday Feb 15, 2008

Sometimes I wonder why I'm in the field of storage. Its not glamorous. Its JBODs, RAID arrays, HBAs, expanders, spinning rust, and all of those things wrapped into enclosures with lots of fans humming. My background is varied, I wrote a file system for my Master's, I worked on one of the biggest Java Business Frameworks ever (the SanFrancisco Project at IBM), and I've danced between the application and infrastructure space more than once.

I often think about my "ideal" job, I've even pondered it here on my blog...and take note, the new Jack Johnson CD is very good and I am ripping it to 8-track real soon now. Personally, I love the field of digital preservation, XAM is in the right direction, and long term digital archives are important to people-kind.

But still, this storage business, there is something to it.

I watched my friend get their eyes lasered to correct their vision this week. While I was watching, I was able to sit with one of the assistants and pepper her with questions, it is an astounding process. Basically, as I understand it, the Doctors use the scanners and computers to

• map the surface of each eye
  
• analyze the surface to understand why the vision is incorrect
  
• create several corrective treatments
  
• the doctor looks at the corrective treatments and adds their wisdom to make the right decision (a lot goes into this, like the health of the patient, the age, their profession, whatever...)
  
• the doctor may tweak the map of places that need adjustments
  
• the updated map is loaded into the "laser"
  
• the patient comes in, gets prepped, the doctor aims the laser and sets the program loose
  
• the "laser" jumps around the eye zapping away
  
• the doctor reassembles the eye
  
• the patient goes home
  

Coolness. But then the geek in me took over, I asked what I could about the machine, backup generators, power, moving the data, mapping the eye, etc... But my head kept thinking about the storage and computer software.

What if a bit is wrong? What if the bits are stored away but due to some battery backup cache being down, it doesn't really get stored and the out of date map is actually in place? What if one tiny point "ages" and becomes rust and there is no checksumming to see it "rotted"? These are people's eyes, you know? Would you want to be the storage vendor that supplied storage that messed up someone's eye because you didn't get the signal / noise ratio on the cabling right?

I've been thinking a lot about digital photography lately as well. While its not people's eyes, it is still an incredibly fragile process. In fact, many of the world's best photographers still do not use digital, and for very good reason. Even when you purchase photographs, you pay a premium price for pictures that have not gone through the digitization process.

Think about this, if a person takes a picture, the CCD (or whatever they are these days) takes the light and transfers it to a memory card. The memory card gets transferred to a laptop hard drive (in my case), a variety of backups are made and I move many of the pictures to SmugMug.

That's a lot of storage along the way. Now, let's say (God forbid), my house burns down. I get my pictures back from SmugMug and one of my pictures has a bit that rotted away.

Now, that is one tiny bit of imperfection to some people. To a professional, that picture is no longer an original. At that point, you have to decide to toss away your artistic integrity and photoshop the point to be like the ones near to it, or just toss the picture from your portfolio. Either way, the picture is never the same.

How would you like to be the one that sold the storage unit that allowed the bit to rot or be stored incorrectly, or archived incorrectly and destroyed that person's memory, that one perfect picture that was meant to be a keepsake forever.

Well, when you think about it, building storage units and management for those storage units is probably not as glamorous as owning the software or companies that specialize in photo archiving, or "lasering" people's eyes, or storing original recordings for artists, or archives of space travel. But those folks have to pick storage units from a company...and if you are the company they pick and you fulfill your moral responsibility to supply checksumming in your file systems, and well-tested storage that may occasionally be late to market to ensure that a memory is not lost or an eye doesn't get fried...you know, that's pretty rewarding.

Cheers to all of my co-workers at Sun who believe storage is more than a spinning drive or a paycheck.