Tuesday Aug 04, 2009
The Sun Storage 7000 Family of Appliances breaks ground in manageability and transparency through an amazing amount of analytics information provided to administrators as well as a highly customizable and extensible management environment that resides on the system. The "Workflow", delivered in the latest release of appliance software, is of particular interest to those of us responsible for "integrating" the Sun Storage 7000 into a management ecosystem, bundling pieces of management logic for use by our peers and reproducing management logic (such as configuration and environmental setup) on several systems at a time.
A workflow is a parameterized piece of logic that is uploaded to a Sun Storage 7000 where it remains resident and is then run via the BUI, CLI or remotely via a shell. The logic within the workflow is programmed in JavaScript (resident on the Sun Storage 7000) and interacts with the system's management shell via "run" commands or built-ins that interact with the current administrative context.
A workflow can do anything that an administrator could do via the CLI, but in a nicely bundled and parameterized way. Here are a few things I've done with workflows:
- gather information about the appliance and reformat it to make it digestable by a higher-level tool
- retrieve sets of analytics data and turn them into different sized chunks (instead of 1 second interval give me a 60 second interval as an average as well as the min and max during the interval) and reformat it to make it easy to digest
- manage the lifecycle of shares (create, manage settings and delete) that are common across appliances
- manage network settings
- create a set of worksheets on every appliance in the network
The opportunities for automation are endless, only bounded by the needs of the administrator in their efforts to integrate the appliances within the management ecosystem.
There is substantial documentation on the appliance's Help Wiki, but for clarity, here is a very simple workflow that will list the attribute of a filesystem that is given as input to the workflow:
- Input: attribute name (same as the attribute in the CLI)
- Output: CSV format: project,sharename,attribute (one line for each share)
- Behavior Notes: a listed attributed that is not valid will return NA in the column (this could be moved to parameter verification but will serve to illustrate exception handling). Also, there are some properties that return empty values as the value was actually inherited from the project context.
Since this is a relatively "short" example, I will simply put the code here with comments and then add additional information afterwords. Note the use of JavaScript functions (such as printToString) as well as the most important element, the definition of the variable "workflow".
/* The printed headers, one will be added with the property name */
var headerList = new Array(
"Project",
"Share"
);
/* A function to print the array into a string for display */
function printToString(csvToPrint){
var csvAsString = "";
for(var i=0 ; i
csvAsString = csvAsString + csvToPrint[i];
// do not finish with an end of line marker
if(i!=csvToPrint.length-1) csvAsString = csvAsString + "\n";
}
return csvAsString;
}
/* This is a required structure for the workflow, it identifies the name, parameters
and the function to execute when it is run */
var workflow = {
name: 'Get Filesystem Attribute',
origin: 'Sun Microsystems, Inc.',
description: 'Prints a Property for all Shares',
parameters: {
property : {
label: 'Filesystem Property',
type: 'String'
}
},
execute:
function (params) {
// prepare the output arrays
var csvContents = new Array();
var currentRow = 0;
headerList[2] = params.property;
csvContents[0] = headerList;
currentRow++;
// go to the root context to start navigation
run('cd /');
run('shares')
// get a list of all of the projects on the system
var projects = list();
// navigate through each project
for(var i=0 ; i
run('select '+projects[i]);
// get a list of all shares
var shares = list();
// go into the context of each share
for(var j=0 ; j
run('select '+shares[j]);
var filesystem = true;
var mountPoint = "";
try {
mountPoint = get('mountpoint');
} catch (err) {
// will end up here if "mountpoint" does not exist, not a filesystem
filesystem = false;
}
if(filesystem) {
var currentRowContents = new Array();
currentRowContents[0] = projects[i];
currentRowContents[1] = shares[j];
try {
var propertyValue = get(params.property);
currentRowContents[2] = ""+propertyValue;
} catch (err) {
currentRowContents[2] = "NA";
}
csvContents[currentRow] = currentRowContents;
currentRow++;
}
run('cd ..');
}
run('cd ..');
}
var newCsvAsString = printToString(csvContents);
return (newCsvAsString);
}
};
While the bulk of the example is standard JavaScript, the workflow structure is where there must be adherence. Here are the important properties:
- name - The name that the workflow will be identified by within the BUI or CLI
- origin - The author of the workflow, can also be used to minimize name collisions
- description - A description of the contents of the workflow, displayed in the BUI or CLI
- parameters - A list of parameters with types (the types supported are listed in the documentation)
- execute - The function that gets executed when the workflow is run (there are more advanced ways of identifying the execution code than are shown here)
The code itself interacts with the system to get a list of the projects on the system, then a list of the shares within the system. The mountpoint property is ONLY available on filesystems, so we know if there is a property error that we do not have a filesystem and skip processing of it (it is most likely an iSCSI LUN).
To upload the workflow, cut/paste the text above and put it in a file. Log into a Sun Storage 7000 Appliance with the latest software and go to Maintenance / Workflows. Click the "+" sign to add a workflow and identify the location of the file. The syntax is error checked on upload, then you will see it listed. Workflows can also be uploaded from the CLI.
Here is what a run of the workflow from the CLI looks like:
isv-7110h:maintenance workflows> ls
Properties:
showhidden = false
Workflows:
WORKFLOW NAME OWNER SETID ORIGIN
workflow-004 Get Filesystem Attribute root false Sun Microsystems, Inc.
isv-7110h:maintenance workflows> select workflow-004
isv-7110h:maintenance workflow-004> ls
Properties:
name = Get Filesystem Attribute
description = Prints a Property for all Shares
owner = root
origin = Sun Microsystems, Inc.
setid = false
isv-7110h:maintenance workflow-004> execute
isv-7110h:maintenance workflow-004 execute (uncommitted)> ls
Properties:
property = (unset)
isv-7110h:maintenance workflow-004 execute (uncommitted)> set property=space_total
property = space_total
isv-7110h:maintenance workflow-004 execute (uncommitted)> commit
Project,Share,space_total
AnotherProject,NoCacheFileSystem,53928
AnotherProject,simpleFilesystem,53928
OracleWork,simpleExport,53928
TestVarious,filesystem1,53928
default,test,448116
default,test2,5368709120
isv-7110h:maintenance workflow-004>
While the example is simple, hopefully it illustrates that this is the start of workflow capabilities, not the entirety of them. The workflow can create management structures (like new shares and worksheets), delete them, modify them, and even enable and disable services.
Workflows make the Sun Storage 7000 an Administrator Development Platform. Try it out in the Sun Unified Storage Simulator if you don't have an appliance at your fingertips!
Sunday Jul 19, 2009
July 17th, I woke up and watched my friend's facebook page turn into a memorial to the great and loving husband that he was. I watched my other friend's page turn into an outpouring of friendship, stunned sadness, and finally...offerings to join her IRL. The effectiveness of a social network for connecting is obvious, the depth of those connections is a bit more mysterious. In the background of the connecting a more human network is coalescing ... one made of phone conversations, emails to coordinate activities, and finally, traveling to build a village around my friend.
The whole episode made me think again about where social networks ... the online variety ... fit in life.
What good are 100 'friends' on a social network if even one of those friends inhibits you from being yourself and really connecting to the people that know you and want to be there for you?
Intertwined social networks become emotional nightmares as people reach out to friends in coded messages while trying to maintain professional coverings. Wrecked relationships, depression, friending and unfriending, bad days ... all of this conducted on tiptoes because peers, bosses, and coworkers are watching over your moves, your playlists, your pictures.
And now I go back to my friend who remains my social network friend but is no longer on our little blue planet.
157 friends on facebook and the only reason this day is anything but an empty mess is because I have the luck of surfing in Hawaii with my boy ... Who doesn't even have an fb account ... And an old (well, she's young ...) friend from Minnesota that delivered her phone number as soon as she connected the subtly coded status message.
So, here's the thing ... Social networks connect ... But they don't replace ... They don't even give insight into where a person's soul is unless that person is willing to risk their social standing at work or was smart enough to keep their work and life networks separate...after all, does your boss really need to know that you're depressed, let alone the reason ...
So...the time has come for me...time to tighten up the social network so my friends can be my friends and let it all hang out ...time to have a beer with a friend IRL ...and time to re-grow that LinkedIn network for the professional side of life.
And ... For my friend ... I'm sorry we didn't have more time here ... Perhaps we will have a chance somewhere else but if we don't ... Well ...
Tuesday Jul 07, 2009
As far back as I can remember I've been fascinated by the ability to move from location to location to get my work done as well as being able to move from device to device. Horizontal mobility (the ability to move across geography) via virtually ubiquitous wireless Internet access is a reality.
This blog post originated in my house, continued between programming sessions at the Highlands Ranch Public Library and then over lunch at The Corner Bakery with only periodic "sleep" modes on my laptop disrupting the work. Horizontal mobility and the ability to work and communicate from anywhere in the country (and often overseas) is so effective, our customers, peers and even our management hierarchies in many industries may not even know where you are geographically located day to day. Have you ever had a conversation start with "Where are you working from?".
Horizontal Mobility shouldn't be news to anyone, I even carry a 3G modem with me so I can work in the infrequent times that I can't find a wireless network. But that is getting more and more rare, consider I had two "Facebook Friends" update their status from an airplane this week.
While horizontal mobility is in our lives to stay, what about Vertical Mobility ... what I would call the ability to move from device to device to achieve a task or handle content? I should be careful to note, when I say device to device, I mean wildly different classes of devices, such as a phone and a laptop, or an Amazon Kindle, a phone and a laptop, or a TV and a computer and a phone. My assumption several years ago, almost a decade now, was that we would move away from the desktop computers and at some point our kids would simply have purpose-built devices giving access to appropriate content for the device ... I jokingly called this a "Fully Integrated Lifestyle" to a few friends the other week but the more I think about it, that is truly what Vertical Mobility is all about.
Are we there yet?
Absolutely. Over the past few weeks I bought an Amazon Kindle and an iPhone so I could revisit the mobile lifestyle. I last visited it about 3 years ago as I architected a Sun Storage Management Portal. At that time, I was playing with re-rendering portlets from HTML / XML to WML. Basically, I would create alternate views of information based on the device being used to view the information (MVC, Model View Controller).
Today, our devices are so powerful and usability of the devices has hit such a high level that alternate views of information are often not required (though I would argue that good vertical mobility does take into account the device). The image above is a fully dynamic view of our Sun Storage 7000 Appliance taken from my iPhone VPN'd to work.
Horizontal and Vertical Mobility
What other devices can be combined? My favorite for vacation is taking TV shows from my Tivo (like Spongebob Squarepants) and moving them to the iPod so my son can watch them while we are stuck in an airport.
The Amazon Kindle has even joined the party. I purchase books on my Kindle and can read them on my iPhone via the Amazon Kindle downloadable iPhone application.
It is truly an amazing world we live in. What's the infrastructure at work here? The cloud is here, the network continues to be built out through our carriers and most of all, content providers that are becoming more aware of the opportunities that abound in content delivery to these devices and, of course, standards around content structure, authentication and authorization, and every other technology that gives us Horizontal and Vertical Mobility and the amazing ability to live a Fully Integrated Lifestyle.
Ok, times up ... back to my design document ... I'll work on it from my Mac today 
Friday Jul 03, 2009
I can summarize my thoughts on trees pretty succinctly, I love a good tree. And with that in mind I finally made the leap and bought my Amazon Kindle DX.
I can safely say that I am holding onto a little bit of the future, it is incredible. The things that drove me over the top to get the Kindle are:
- I am VERY schizophrenic in my reading enjoyment, sometimes having as many as 4 books for reading open, 2 or 3 technical documents, and several large PDFs full of APIs
- I received The Denver Post Thursdays through Sundays and didn't read it most of those days ... but never knew when I wanted it
- I go through spurts of travel and I ALWAYS have the wrong book with me ... my moods swing notoriously when I'm traveling and what I want to read or learn about swings with it
- I get tired of lugging power cords everywhere
And so my great Kindle DX experiment began, just about Father's Day (my gift to myself for being a single Dad with little or no time for book enjoyment).
First off, it is very, very easy to spend money with it ... albeit there is a discount on most books that you purchase for the Kindle. Oddly, this doesn't hold true for most Computer Books, they seem to be full price or near full price across the board. I've already purchased and read several books, including the amazing Darwin's Radio and Darwin's Children books from Greg Bear (conveniently in one "book" for the Kindle) and am now working on The Road and Anathem from Neil Stephenson. All bought sitting, literally, from the comfort of my bed or while camping. I have also downloaded quite a few work PDFs and presentations to it and receive The Denver Post electronically each day, automatically delivered.
Here's a hint on Anathem, I bet my back doesn't hurt from lugging it around and yours does ... and guess what, I have a built-in dictionary so I can just move the cursor to a word and the definition appears on the bottom of the screen.
The downside of newspapers is that they are difficult to read and don't include the two best parts ... the comics and the ads. As such, I still recieve a Sunday Denver Post ... but my recycling bin has definitely went down in size.
When folks ask me about it, I put my Kindle experience rather succinctly:
- Book Reading (fiction / non-fiction): A+ (as easy on the eyes as paper, easy to navigate, etc...)
- Reference Literature that you Skip Around in: C (VERY difficult to skip around a book and "leaf through" the pages
- Newspapers: B-
- Ease of Use: B+
- Ease of Shopping: B (I usually shop on a web browser rather than the built in store but if you know what you are buying, the built in store on the Kindle is easy enough)
- Cost: C- (this has GOT to come down and I completely object to paying for blog subscriptions ... sorry Amazon and Slashdot, ain't gonna pad your profit margins)
- Ease on the Eyes: A+ (as good as paper and my eyes don't get all wobbly like on a computer...I was literally reading under the covers so-as to not annoy my son at Scout Camp...and it's not backlit so I had my flashlight with me, it was just like when I was a kid!)
- Portability: A+ (I have cut down the size of my backpack by half ... I still have to carry my Mac)
The power lasts FOREVER on the thing. Even with 3G on I plug it in at most once a week.
Now here is the BIGGEST complaint I have. Flying to San Francisco they SPECIFICALLY point out that your Kindle must be turned off. People, it barely uses power. Might I suggest you just ask folks to turn off the 3G on the Kindle instead?
I have even put all of our docs for the Sun Storage 7000 Appliance onto it for quick access at customer black or gray sites. Very handy to have along.
Now here is a tip for those of us who carry these around, you will have people tell you straight out that you are a heretic for abandoning "the feel of paper" and contributing to the "demise of the book" (Auntie, if you are reading this...this is for you . It is SIMPLY not true. In fact, there are a few things that actually seem to work for the author:
- There is no after-market for a book, this is frustrating to me but as long as most books are discounted I am willing to deal with it for now. But consider, I can't resell a book that is on my e-reader and I understand that this is a slippery slope to create.
- Personally, books, PDFs, and stuff like that contribute to about 30% of the clutter in my house and frankly, I'm not one to enjoy having bookshelves of the things I've read contributing to the dust in my environment...they are pretty much g-o-n-e
- I've already saved multiple trees through e-news delivery and Anathem
But in the end, it's not for everyone. But after the end, the demise of the paper novel is nigh my friends, it may not be my generation but I guarantee that my son will be 50% e-books at least in his High School days, by college 75% and his children will not buy paper books, end of story.
I notice this post is exceeding the expected length, drop me a line if there is anything you want to know about!
Sunday Feb 08, 2009
I was just browsing the agenda for the OpenSolaris Storage Summit that is being held on February 23rd 2009 in San Francisco (city with that crooked street ya know).
It turns out one of the big talks is going to be from Don McAskill from SmugMug ... and I kid you not and I'm not doing any schmooozing here, but it is my favorite favorite photo sharing site so I've already signed up. Also on the list are Mike Shapiro or Randy Bias from GoGrid. What does Storage and OpenSolaris have to do with the cloud????
It turns out, everything in the cloud has to end up SOMEWHERE and if its not using the cloud itself as a storage utility (via the likes of Amazon S3, GoGrid and others), the cloud applications themselves use a boatload of storage. The Sun Storage 7000 family and the Sun Open Storage J4000 family coupled with OpenSolaris are definitely the way to go for infrastructure.
Doh, I'm totally geeked and off to the travel site to figure out how to book my trip and get my kiddo taken care of for the day (though as you can see...he loves San Francisco).
Thursday Feb 05, 2009
In the storage industry (and the computer industry in general) we have this term known as a "dissatisfier". A dissatisfier is basically a feature or a product that people have to use that no matter how well you do it, you will almost never receive a "win" (or even an acknowledgment of how well you implemented the feature) since the customer isn't really buying the product for that feature. On the other hand, if you implement the feature poorly, you can lose deals and cause a great deal of consternation on customer sites.
Solaris 10 Update 6 (as well as the latest OpenSolaris release) fixes a huge dissatisfier for me. After choosing what devices to install the system on, in previous versions of Solaris, you were met with a prompt asking if you wanted to partition the storage by default or customize the directory allocations. At this point you would go in and choose how much storage goes into the root (/), how much goes into home directories (/export/home), how much is swap and you could add your own partitions. I know I'm bad, but since I'm an engineer I never use /export/home on my system so I have to jigger the directories and fix how much storage is allocated.
Along comes ZFS Boot. Now, there are a LOT of reasons to love ZFS boot (we use it on the Sun Storage 7000 and on most of our developer systems). On previous projects, we noticed how quick and easy it was to do things like LiveUpgrade.
Now, remember that ZFS is natively thin provisioned...meaning, you can create file systems that all sit over the same pool of storage (multiple drives, devices, enclosures) and each file system vies for the storage (you can add quotas and reserve space, don't worry).
Today I installed Solaris 10 U6 and blew through the installation process without EVER having to finagle a screen saying how much storage should be allocated to each directory. YAAAAYYYYYYYYYYYYY
Basically, you choose to install with ZFS, select two disks (so the install can be mirrored and one device can fail) and off the install goes happily ever after.
Ahhhhh, its the little things ....
Thursday Jul 17, 2008
After years of trying to squirrel away a few minutes to install my only BitTorrent swarm in our lab, I've finally done it! I thought I'd put the instructions out here on the web since it seems that explicit Solaris-based instructions are relatively sparse (they are sparse because the version I'm using is relatively easy to use and not relegated to a single platform...thus the lack of Solaris-specifics).
I started with a clean build of Solaris 10 Update 5 on one machine, and Build 94 of Nevada on another set of 4 machines. These are the Sun Fire x4150s I've discussed in other posts.
Retrieving all of the packages I needed was straightforward with Blastwave.org (an open source software repository for Solaris). In fact, with just a few commands I had a complete BitTorrent stack installed that runs with Python (don't have Python on the system...Blastwave takes care of that too).
Install BitTorrent
First, read the HOWTO on using Blastwave. I only made it through a few steps of the HOWTO before I was up and running, so don't sweat it if it seems long and you are attention-challenged (like me without a special Nespresso Level 8 or above espresso). Of course, I'm fumbling around more than I should so be sure to read the whole thing when you do have time, it contains information on directories and where stuff gets put (at least I think it does...). Installing what you need to use Blastwave takes only a few minutes. Once you have Blastwave ready and running, retrieve BitTorrent and all its dependencies (primarily Python):
/opt/csw/bin/pkg-get -i bittorrent
That's it, BitTorrent is now installed in /opt/csw/share/BitTorrent
Remember the BitTorrent Components
Here is the 2 minute review of important BitTorrent Network Components and terminology that follows the general deployment diagram here (this is the swarm I've installed in my lab so I'll refer to it throughout):
- Torrent - A "file" that is prepared to be distributed by BitTorrent, torrent may also refer to the metadata file prepared from the contents of the file you will be distributing
- Tracker - A "tracker" helps refer downloaders (peers) to each other, sort of an organizer of nodes
- Swarm - A group of peers that are delivering content to a requester (the downloader itself often becomes a part of the swarm for another requester very early in the download process)
- Seed - A full copy of a file that clients can obtain. This seems like a heavy load on a single server, but very early on clients will lessen their load on the Seed systems and depend on other peer clients for chunks of the torrent.
In the scenario below, the Tracker and the Seed for my torrent (OpenSolaris 08.05 ISO) are one and the same. The Seed should typically be split from the Tracker though...the Tracker is a bottleneck and you don't want it to be pounded on by Peers.
Start a Tracker
We will start the tracker first. A tracker can be started as a background task. It will sit on a port and listen for requests and write output to a file.
./bttrack.py --port 6969 --dfile dstate --logfile tracker.out
Your tracker is now up and running and ready to organize clients.
Create a Torrent
A "torrent" metadata file contains all of the information about a particular file that can be retrieved through BitTorrent. The torrent is associated with a tracker and when complete will contain a variety of information to ensure the integrity of the final torrent when a client receives it.
The README.txt with the BitTorrent download didn't have proper instructions for building a torrent (seemed a little out of date), so be careful and don't get frustrated. Use the btmaketorrent.py command to build the torrent file. I've chosen to create a torrent from the OpenSolaris 08.05 distribution (my favorite distribution of all, of course...):
./btmaketorrent.py --comment "OpenSolaris 08.05 from Pauls Swarm" http://x4xxx-01.sun.com:6969/announce os200805.iso
Notice the use of the tracker that I set up. You can show the meta information that was generated by using the btshowmetainfo.py command:
# ./btshowmetainfo.py os200805.iso.torrent
btshowmetainfo 4.0.4 - decode BitTorrent metainfo files
metainfo file.: os200805.iso.torrent
info hash.....: fdf239d2524e44432892d01ab354e20a8b77b7e6
file name.....: os200805.iso
file size.....: 719087616 (2743 * 262144 + 26624)
announce url..: http://x4xxx-01.sun.com:6969/announce
Setting up a Seed
You need a web server available from the system that is going to be a "Seed". Nevada has Apache 2.2 (at least in my build) installed by default, you simply have to turn it on (mileage may vary). You can do this by typing:
# svcadm enable apache22
# svcs | grep -i apache2
online 20:41:36 svc:/network/http:apache22
A better instruction set can be found on BigAdmin.
With a little nosing around the system, I found my httpd.conf file at /etc/apache2/2.2. Look in the file for the DocumentRoot, something like this works:
# cat httpd.conf | grep DocumentRoot
# DocumentRoot: The directory out of which you will serve your
DocumentRoot "/var/apache2/2.2/htdocs"
We will place our .torrent file (created earlier) in the DocumentRoot.
Most web servers you also have to associate the mime type "application/x-bittorrent" with the file extension "torrent", this was already done for the pre-installed Apache 2.2 on Nevada.
Finally, put the complete file on a server (the same one would work like I am doing, but a separate server is recommended). Run the BitTorrent download, saving the file into the location that the file already exists. BitTorrent is smart enough to see all of the chunks are there, but it spends time verifying the are correct. When complete, the computer you run the download is a Seed. This command works for the OpenSolaris 08.05 torrent I created earlier:
# ./btdownloadheadless.py --url http://x4xxx-01.sun.com/os200805.iso.torrent --save_as os200805.iso
This takes a while as it runs through hash checks on the existing file.
The torrent is now ready as a Seed for peer to peer access. You can add as many Seeds as you want depending on how popular you think your file is going to be.
Use another Client to Download a File
I can now go to x4xx-02 and start the headless download, just as the previous one ran.
# ./btdownloadheadless.py --url http://x4xxx-01.sun.com/os200805.iso.torrent --save_as os200805.iso
Remember the torrent itself points back to the tracker. There should be some brief activity by the tracker during the download to see if other peers in the swarm can help.
During the download, you will become a part of the swarm from which other clients can download chunks of OpenSolaris (you have to give it, if you want to get it - The Suburbs).
As the file starts to download, you will see the transfer rate start to go up. With a single Seed, I started slowly ramping up. BitTorrent is careful to balance requests. Remember, I now have a Seed and a Peer with chunks of the file. With a single download in progress, I reached about 20 KB/sec (the upload from the Seed and the download from the client)...there is some obvious throttling going on somewhere.
What if I start a download of the torrent on x4xxx-03? The Seed remained uploading around 20 KB/sec. After an initial hit on x4xxx-02 down to about 14 KB/sec, I quickly moved back to over 20 KB/sec, while x4xxx-03 was peaking near 30 KB/sec. As more chunks moved onto x4xxx-03, x4xxx-02 also sped up, since it could grab chunks from two peers. The Seed remained constant at a 20 KB/sec upload, but x4xx-02 was now also uploading at 20 KB/sec and x4xxx-03 was able to have the aggregate bandwidth.
But wait, I have two more clients sitting idle, I started up x4xxx-04 and x4xxx-05 with the download. Again, an initial hit from x4xxx-02 occurred, as it was heavily relying on the original Seed and clients were also relying on it. Within about a minute, the original Seed was still uploading at 20 KB/sec, but -02, -03 and -04 were also uploading at 20 KB/sec. All clients were now downloading at 20 KB/sec with the original seed still uploading at a constant 20 KB/sec...peer to peer amortization of upload bandwidth at its finest.
As the download moved along, more chunks flooded onto the remaining peers and the sharing became much more efficient. Each of the clients regularly found their way over 25 KB/sec and often into the 30 KB/sec range.
I decided that the graphical output wasn't very fun, so I added my home system that I VPN with into the party using Azureus. With Azureus, I'm able to get graphical displays of what's going on. Here is a picture of my swarm:
Note that the center circle is my client, the fully blue circle is the seed, and you can see the other peers don't yet have a lot of chunks of the file.
Here is another interesting view from Azureus:
Of particular note is the Swarm Speed, we have hit about 100 KB/sec with our 5 peers and an obvious upload throttling on each client at around 20 KB/sec. Well, I'll obviously be looking into that...but I think I've run out of words for the evening and I've more than achieved my purpose tonight. Enjoy BitTorrent on Solaris!
Now, I just have to figure out what other kinds of fun I can have with my swarm
Thursday Jun 26, 2008
I've been working back in the Auto Service Request space for the past few weeks and doing some education and stuff as we build and test this product or that, it seemed like a great opportunity to put out a quick and dirty "SNMP with the FMA" blog post. There are some excellent previous materials out there that are extremely applicable and you could probably get by with, so first I will point those out:
With those articles and a little help from your nearby SNMP guru, you are pretty much good to go. I've extended the information in those papers a tiny, tiny bit with how to link the information back to Sun's message repository. The great thing about the FMA is that we have "special sauce" around that can create a full-service cycle (including a Remote Shell application that service can use with your permission and oversight to further diagnose problems from a remote location as well as tools to deliver faults back to Sun Microsystems, including our own Common Array Manager).
Introduction
This guides a user through a 2 system setup with
- system1 - enabled to send traps when a fault is detected by the FMA
- system 2 - enabled to receive traps from system1
Throughout this write-up, a roughly equivalent FMA command is given that can be run on the originating host so you can follow what the SNMP trap is delivering. To me, the FMA command on the originating host is definitely the preference though, since it usually provides additional information and formatting that may not be available in the SNMP trap or walk.
Setup of Trap Sender
system1 must be setup to deliver traps when FMA events occur.
To do this, follow the instructions provided at A Louder Voice for the Fault Manager, summarized here:
- Create the file /etc/sma/snmp/snmp.conf (if it doesn't exist) and add the line "mibs +ALL"
- Add the line to /etc/sma/snmp/snmpd.conf: trap2sink system2
- Add the additional line to support deeper information query: dlmod sunFM /usr/lib/fm/amd64/libfmd_snmp.so.1
Assuming you have made no other changes prior to this, your system should be ready to go. You do have to restart the SNMP service at this point (I always reboot...its a Windows habit).
Setup of Trap Receiver
system2 must be setup to receive traps, this is simple for demo purposes:
- Run /usr/sfw/sbin/snmptrapd -P
- Watch for traps
Receiving FMA traps
When system2 receives FMA traps from system1, they will look like this (formatting appropriately rearranged)
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1190862249) 137 days, 19:57:02.49
SNMPv2-MIB::snmpTrapOID.0 = OID: SUN-FM-MIB::sunFmProblemTrap
SUN-FM-MIB::sunFmProblemUUID."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"
SUN-FM-MIB::sunFmProblemCode."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: ZFS-8000-CS
SUN-FM-MIB::sunFmProblemURL."33570acb-e108-4ca8-8276-c67aeecf2043" = STRING: http://sun.com/msg/ZFS-8000-CS
Check out that URL in the sunFmProblemURL, http://sun.com/msg/ZFS-8000-CS. You can actually go there and get the System Administrator Actions that should be taken as well as an extended view of the problem (not contextualized to your view, but this is the general fault information that you would see if you were on the system itself).
This trap is roughly equivalent to the information you would receive from running the
basic fmdump command on the system with the fault. You could also run the "fmdump -v -u 33570acb-e108-4ca8-8276-c67aeecf2043" command on the trap originator to get a bit more information:
bash-3.00# fmdump -v -u 33570acb-e108-4ca8-8276-c67aeecf2043
TIME UUID SUNW-MSG-ID
Jun 26 10:57:45.3602 33570acb-e108-4ca8-8276-c67aeecf2043 ZFS-8000-CS
100% fault.fs.zfs.pool
Problem in: zfs://pool=data
Affects: zfs://pool=data
FRU: -
Location: -
Navigating Problems on the Remote System
Let's start at the top now. We can quickly navigate all known, unresolved problems on the remote system by walking the sunFmProblemTable:
system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmProblemTable
This results in a dump of all problems with the UUIDs that can be used for deeper queries. The following printout is only a single problem on the system, additional problems are listed in blocks by the attribute (so all sunFmProblemUUIDs are lumped together followed by all ProblemCodes, and so on).
SUN-FM-MIB::sunFmProblemUUID."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: "0f3dcdf3-f85b-c091-8f1f-ce2164976cda"
SUN-FM-MIB::sunFmProblemCode."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: ZFS-8000-CS
SUN-FM-MIB::sunFmProblemURL."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: http://sun.com/msg/ZFS-8000-CS
SUN-FM-MIB::sunFmProblemDiagEngine."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: fmd:///module/fmd
SUN-FM-MIB::sunFmProblemDiagTime."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = STRING: 2008-6-18,10:7:4.0,-6:0
SUN-FM-MIB::sunFmProblemSuspectCount."0f3dcdf3-f85b-c091-8f1f-ce2164976cda" = Gauge32: 1
This is roughly equivalent to the fmdump command on a system, though with the basic fmdump only the UUID, Code and a MSG-ID are given.
Based on this information, we can look at the ZFS-8000-CS message at sun.com and determine what our next steps should be. It indicates that using the zpool status -x on the system with the fault would be useful. Going to the originating system and running it returns:
bash-3.00# zpool status -x
pool: data
state: UNAVAIL
status: One or more devices could not be opened. There are insufficient
replicas for the pool to continue functioning.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-D3
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
data UNAVAIL 0 0 0 insufficient replicas
mirror UNAVAIL 0 0 0 insufficient replicas
c2t0d0 UNAVAIL 0 0 0 cannot open
c3t1d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t2d0 UNAVAIL 0 0 0 cannot open
c3t3d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t4d0 UNAVAIL 0 0 0 cannot open
c3t5d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t6d0 UNAVAIL 0 0 0 cannot open
c3t7d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t8d0 UNAVAIL 0 0 0 cannot open
c3t9d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t10d0 UNAVAIL 0 0 0 cannot open
c3t11d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t12d0 UNAVAIL 0 0 0 cannot open
c3t13d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t14d0 UNAVAIL 0 0 0 cannot open
c3t15d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t16d0 UNAVAIL 0 0 0 cannot open
c3t17d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t18d0 UNAVAIL 0 0 0 cannot open
c3t19d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t20d0 UNAVAIL 0 0 0 cannot open
c3t21d0 UNAVAIL 0 0 0 cannot open
mirror UNAVAIL 0 0 0 insufficient replicas
c2t22d0 UNAVAIL 0 0 0 cannot open
c3t23d0 UNAVAIL 0 0 0 cannot open
bash-3.00#
For some history on this particular problem, we disconnected a JBOD that had the "data" pool built, so none of the devices are available...ouch.
You can look more deeply at the events that resulted in the problem by walking the sunFmFaultEventTable:
system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmFaultEventTable
SUN-FM-MIB::sunFmFaultEventProblemUUID."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: "0f3dcdf3-f85b-c091-8f1f-ce2164976cda"
SUN-FM-MIB::sunFmFaultEventProblemUUID."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"
SUN-FM-MIB::sunFmFaultEventProblemUUID."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: "3600a05e-acc1-cae2-c185-f50852156777"
SUN-FM-MIB::sunFmFaultEventProblemUUID."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: "97bfeb63-7b02-c2b6-c51f-c451a9f760c5"
SUN-FM-MIB::sunFmFaultEventClass."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventClass."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: fault.fs.zfs.pool
SUN-FM-MIB::sunFmFaultEventCertainty."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."33570acb-e108-4ca8-8276-c67aeecf2043".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."3600a05e-acc1-cae2-c185-f50852156777".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventCertainty."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = Gauge32: 100
SUN-FM-MIB::sunFmFaultEventASRU."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: zfs://pool=1ca09fa50e7ca8c7
SUN-FM-MIB::sunFmFaultEventASRU."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventASRU."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: zfs://pool=6f658a6c4b99b18b
SUN-FM-MIB::sunFmFaultEventASRU."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventFRU."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventFRU."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: -
SUN-FM-MIB::sunFmFaultEventResource."0f3dcdf3-f85b-c091-8f1f-ce2164976cda".1 = STRING: zfs://pool=1ca09fa50e7ca8c7
SUN-FM-MIB::sunFmFaultEventResource."33570acb-e108-4ca8-8276-c67aeecf2043".1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmFaultEventResource."3600a05e-acc1-cae2-c185-f50852156777".1 = STRING: zfs://pool=6f658a6c4b99b18b
SUN-FM-MIB::sunFmFaultEventResource."97bfeb63-7b02-c2b6-c51f-c451a9f760c5".1 = STRING: zfs://pool=data
This is roughly equivalent to the fmdump -v command. The fmdump -V command cannot be duplicated over SNMP, though it can be useful to run on the host side. fmdump -V can provide product, chassis and server IDs as well as a more complete list of faults and ereports that resulted in the diagnosis.
You could also view the fault management information by resource rather than by fault or event over SNMP
system2:/root-> /usr/sfw/bin/snmpwalk -v2c -c public -t 20 system1 sunFmResourceTable
SUN-FM-MIB::sunFmResourceFMRI.1 = STRING: zfs://pool=data
SUN-FM-MIB::sunFmResourceStatus.1 = INTEGER: faulted(5)
SUN-FM-MIB::sunFmResourceDiagnosisUUID.1 = STRING: "33570acb-e108-4ca8-8276-c67aeecf2043"
This is similar to the "fmadm faulty" command that can be run from a system. The faulty parameter results in the some additional information and text, though that text can also be retrieved at the event URL that was earlier identified.
While this is the "SPAM" approach to inquiring about a system, you could also walk each problem as they come in.
As was mentioned previously, there isn't a "ton" of information available within the SNMP MIB itself. The result of receiving a particular trap is often to do additional diagnosis on the system with the fault. So some ssh work may be necessary.
Additional things to do with FMA
You can dump the topology of a system!
Go to /usr/lib/fm/fmd and type "fmtopo", you should get a complete topology as it is recognized by the fault management infrastructure.
bash-3.00# ./fmtopo
TIME UUID
Jun 26 13:20:16 ddba8792-0166-6fb4-a81b-db3de9622649
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=1
hc://:product-id=Sun-Fire-X4150:chassis-id=0811QAR189:server-id=server1/motherboard=0/chip=0/cpu=2
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/chip=0/cpu=3
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0/dram-channel=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0/rank=0
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=518545073102016426:part=7777777:revision/motherboard=0/memory-controller=0/dram-channel=0/dimm=0/rank=1
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1/motherboard=0/memory-controller=0/dram-channel=1
hc://:product-id=Sun-Fire-X4150:chassis-id=09999:server-id=server1:serial=555555555:part=777777777 :revision/motherboard=0/memory-controller=0/dram-channel=1/dimm=0
... (and on and on)
Artificial Fault Injection
There isn't a lot of documentation I've found for fminject, but it looks like it can be quite useful. It will, at its easiest, allow you to replay a set of ereports and faults that previously occurred in a system. For example, we can replay the error log that resulted in the FMA trap for the ZFS pool problem that the trap was sent out for. In one window on the system that sent the trap, start up the simulator
cd /usr/lib/fm/fmd
./fmsim -i
In another window, replay an error log
cd /usr/lib/fm/fmd
./fminject -v /var/fm/fmd/errlog
You will see the SNMP Trap re-appear in the trap receiver. Note there is no good way to determine that this is a replayed trap in the trap receiver's window that I've determined, so use this with caution.
Summary
While there is enough information to completely diagnose a problem from an SNMP trap, the traps are a great opportunity for integration with higher level management tools. More often than not, additional commands and information are required to be run on the system with the fault (be it logs or output of various commands, especially in the ZFS space), but the messages themselves typically have the next steps for diagnosis.
Friday Jun 20, 2008
I build, maintain and pay for hosting for my friend's charity web site, Play for a Heart. For the past several years I've had a host provider where I would deploy the very simple web site I constructed for her. The host had become my "cloud" in the stormy skies of Cloud Computing.
This cloud kept drifting along, the price was right, it was somewhat reliable, it was ... easy.
I went to the web site on a Saturday morning, it was gone. Somewhere over the course of the last few months the company had turned into a shell of the host provider it formerly was. There was no tech response over the weekend and the forums were more or less ghost towns. I made a post and the only responses were "Leave this host as fast as you can". The outage continued on through the weekend and extended to a large percentage of the host's own sales and marketing sites.
Finally, late Sunday I couldn't take it anymore, I pulled the plug and moved to GoDaddy. The move had nothing to do with Danica Patrick. Here was my very simple logic:
- Java Support / PHP Support / MySQL Support
- Able to host multiple domains with a single hosting account
- Resilient enough to support the onslaught after a Super Bowl commercial
- Good recommendations and community
- Great price
And I switched...
Lucky for me I made my friend's site as basic as I could.
- Very simple HTML
- Slideshow objects embedded from another Cloud Application (SmugMug so that the links moved right over (and I only put these on after she had enough sponsors that I couldn't figure out how to keep the sponsor page clean)
- No applications embedded directly from my low-budget hosting solution
- Domain Names purchased from Yahoo and GoDaddy (this became a HUGE win as the host records would have been locked had I purchased the domain names with this smaller company)
My web site was moved to GoDaddy and back up and running in, literally, 1.5 hours and my exit was complete from my previous host.
Still, the entire experience left me very shaken over Cloud Computing. I've come up with the following set of thoughts when it comes to attaching my digital life to the clouds:
- Many companies literally own the information you create or have very liberal rights to that information (Always read the License Agreements)
- Many companies have no exit strategy for your information (once your data is captured or created in the online application it cannot be extracted...this is especially the case in social networking infrastructure)
- Because of the nature of the cloud, you have no guarantees that many companies that host the applications you depend on are even viable (check the business model and financials if you are tying your life to the cloud)
- Several companies entire business models are centered on analyzing the information you give to them and monetizing it (in many cases "personalization" in your eyes is really "targeting and demographics" in the company's eyes)
- Standard Platforms, Standard APIs, Standard Information Models are at the heart of a successful cloud (not fog), this allows you a better chance at having tools to import / export information and interact with that information...and in the case of building an application, it is critical that these standards are the core of the application so that you can get out of your cloud as quickly as you got into it and at the first sign of turbulence
- In lieu of standards (even defacto), Open APIs, Open Source, Open Architecture, and a good open license can be a huge help...especially with a robust developer community. This assures infrastructure can live beyond the life of any individual cloud if it should disappear.
- Buyer beware
Well...I'm a huge, huge advocate of Cloud Computing. Our own network.com site, Amazon EC3 and S3, SalesForce.com, they are all leading the wave...and in 10 years it will just be the way it is done. But until then, there will certainly be some growing pains, probably more in the small-company space than in the enterprise space...but they will be there.
Standard's groups, defacto standards, open source cloud infrastructure, entry and exit for information, information licensing, information security and transparent motivation for collection of your information will go a long way to determining whether there is longevity in the sunny skies for cloud computing.
Thursday May 29, 2008
I am in San Francisco at Sun's Preservation and Archive Special Interest Group (PASIG) Face to Face meeting. Preservation and Archiving is one of the most fascinating and complex problems to face our digital world, in my humble opinion. I've chatted about this before in the blog so I won't belabor the point that taking a chisel to the walls of this beautiful location and etching in the contents of your e-book would have a longer life span than most of our media types in use today.
Still, chiseling has some pretty serious downsides
- It doesn't scale well (oh, and you have to scale to get up the wall)
- You have to travel to this site to view the contents (while that might be pleasant, it can be quite inefficient)
- Its difficult to create a geographically remote backup site
- It seriously blemishes a beautiful location
It turns out, there is a community of people that think about this problem as their day job and PASIG is a gathering point for that community. The community is made up of the "new" digital librarians, software that provides the foundation for digital preservation and archives, storage vendors that can provide "temporal" storage locations for the digital archive, and system integrators that can customize the entire stack and tailor it for a particular domain (astronomy, human sciences, etc...).
The community is global, but their is an obvious disparity in how this type of work is funded across countries and boundaries. Here in the U.S., the leaders of preservation and archiving efforts appears to be the education community, with a variety of colleges (Penn State, University of Pittsburgh (Pennsylvania), etc...) and individual government agencies (Library of Congress, National Archives, etc...). Abroad, it seems the repositories are often nationally funded.
While various technologies were interesting, it was also interesting to hear the oft recurring theme of "sustainability"...how can we "sustain" the rate of ingest and cost over time of these types of solutions? Consider the cost of the initial storage of 1 GB of data today (just the storage medium), its between $0.25 and $1.00 depending on your storage medium. Now add in the cost of the various transitions that data makes over 20,000 years (life of a cave painting) the system administration, the building of new equipment, the power involved, etc... At this stage of human evolution, it is difficult to even predict what type of media we will store data on in 20 or 30 years. How can one predict the cost of sustaining these archives, when we won't know what the archive will look like, let alone the environment around it, in 20 years, 100 years or even 1,000 years.
Which brings up another interesting thread...whereas a cave painting is required to stay on the rock it originated on, the life of preserved digital data (and archived) must be assumed to migrate over the data generations. Consider each generation of media about 4 years (3-5 is common), tape can technically survive multiple generations and one can build systems (like the Sun Storage 5800 (Honeycomb) platform) that can theoretically store data for 100,000s years. But the reality is that the lifetime of a technology and a media is that you migrate approximately every 3-5 years.
This is a conceptually different model then many people think of when they think of an "archive" or a "storage vault". Rather than think of a monolithic system that weathers the years, its important to think of data as migratory, and evolutionary (much like humans themselves). To survive 30,000 years, our digital data originates...often in the physical world...but increasingly our data originates in the digital world itself...and migrates to a Tier 1 storage device. Regardless of whether it remains there, over the next 20 years, that data will move across 4 or more devices (not counting replicants or backups) if it stays "online".
So what are the three most important attributes of an archive and/or a preservation system (IMHO):
- The information is safe, stable, and can be verified to be "original" while it resides temporarily within the system
- The information can be moved off the system when the time is right and can still be certified as "original" data rather than a derivation of the data
- Information can be "moved into" a new preservation / archive system.
In the end, long-term data is migratory...and our systems and software that support this field must start thinking of data that way. This type of thinking can also be applied backwards into more traditional storage paradigms, how useful would it be for a photo repository to be able to migrate to bigger and bigger system with higher capacity and lower power consumption? At some point, there is a trade-off that will trigger that company to move, and facilitating that move is in the best interest of all companies.
What else came up at the conference? Tons of things, have we thought enough about:
- Access Rights (and access across time)
- Roles and Responsibilities in a digital archiving and library maintenance system (the new librarians)
- Geo-political boundaries as repositories move into the "cloud"
At this very minute, Sam Gustman is discussing The Shoah Foundation. The mission of The Shoah Foundation is To overcome prejudice, intolerance and bigotry - and the suffering they cause - through the educational use of the Institute's visual history testimonies. The documentation of The Holocaust contains over 120,000 hours of online video testimony. The foundation indexes minute by minute of this testimony and makes it accessible for people to teach and understand the impacts of the holocaust. The foundation moves beyond The Holocaust into testimonies of other events in human history that must be recorded and understood. As George Santayana said: Those who do not learn from history are doomed to repeat it.
Sam's presentation isn't online, I'm trying to get Art to post it. This work alone shows that digital archiving and preservation is not just an interesting segment of compute and storage...its a moral imperative.
I found this link to a video that discusses Fedora Commons, one of the open source repository software initiatives for creating and managing digital libraries and archives. The video discusses some of the solutions that are being built with the Fedora Commons software solution:
What a great opportunity the conference has been to sit and think about the big picture (and archiving and preservation truly are the big picture).
Thursday May 15, 2008
We announced a whole set of new servers this week, including one of my favorites, the Sun Fire X4240 Server. I only like to talk about things I've touched and played with so I figured I'd show you a little about how I'm using my x4240s.
I'm a storage guy, so what I immediately like about this particular system (and the Sun Fire x4150s I talked about before, as well as the related 8 and 16 disk offerings with SPARC chips), is the disk density I get along with the raw horsepower sitting right next to the disk drives.
The Sun Fire x4240s I'm using right now have 2 x Dual-Core AMD Opteron 2222s that run at 3.0 GHz, but you can have a couple of Quad-Cores in this thing as well! As for storage, I have a set with 146 GB, 10k RPM SAS Drives that are pumping about 66 MB/sec through a single NIC port with our Open Storage CIFS Server that integrates nicely with ZFS.
Watch me share 15 drives through CIFS so those poor Windows folks can attach to my box for their storage (and get the benefits of ZFS snapshot and capacity):
$ zpool create sharepool raidz2 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 raidz2 c1t6d0 c1t7d0 c1t8d0 c1t9d0 c1t10d0 raidz2 c1t11d0 c1t12d0 c1t13d0 c1t14d0 c1t15d0
$ zfs create sharepool/share1
$ zfs set sharesmb=on sharepool/share1
$ zfs set sharesmb=name=share1 sharepool/share1
(You also have to configure the PAM module appropriately, but this is a one time thing, try this:
1. Add the below line to the end of /etc/pam.conf:
other password required pam_smb_passwd.so.1 nowarn
2. Set/Change the Passwords for any Solaris User That Will be Used to Authenticate when Connecting to a CIFS share, like this:
$ passwd pmonday
New Password:
Re-enter new Password:
passwd: password successfully changed for pmonday
With the SMB PAM module installed, passwords are generated that can be used by Windows as well as Solaris (you have to do this). There are, of course, differences from your normal Samba installation but with everything, figure out the tool that is best for you and go with it 
One thing you notice with Sun's CIFS Server is there is no anonymous access to shares (this can be a good thing)!
Sun's CIFS Server and the Sun Fire X4240 are just one example of how the Open Storage initiative pays off. You can turn this capacity / performance rich server into an iSCSI target, NFS target, or even keep the storage to yourself and load applications directly on the box that can make use of the fast SAS spindles. Don't want to use Solaris or OpenSolaris...heck, load Red Hat or Windows on these things, we do it all the time...in fact, we'll even pre-load a Linux-variant for you (check the options page).
Cool stuff, nice job System's Group! Oh yeah, and the iLOM in the x4240 works GREAT (better than the x4150 even).
Friday Apr 25, 2008
For the past week or so I've been on one of those "get this done" projects from my boss. Its tough to "get this done" when you are commuting though (or when the lab is the same temperature and wind speed as the top of Loveland Pass). My project involves 5 Sun Fire x4150s (and will involve another set of servers next week).
I did have to go into the office for a few days to pull memory in and out of them, do some DVD swapping on the machines, check cables, and (of course) get the 8 x 73GB 15K SAS drives swapped in (all of my machines need the same drives and speed).
During the setup, Terry Hill allocated IP addresses for the service processor eLOM (embedded Lights Out Manager) and configured the IP addresses within the BIOS. Keep in mind when you look at the back of a Sun system with eLOM, you see the standard ethernet ports (typically 4 on this size box), but you also see an additional port that's offset from the others and labeled "Service Processor". It is next to the Serial Console port. You can see it here, on the left side of the back of the system.
(The x4150 eLOM documentation has more details about configuring eLOM).
Once the network is configured, you type in the web address of your x4150 service processor (http://myx4150.xxx.com) and up pops the eLOM Web User Interface. You can do this from anywhere (even home
Now that I'm in the eLOM, I can do all sorts of things. I primarily use the Web interface. With it I can do all of the power resets I need to do (including having it come straight up to BIOS). I can also launch a console through the magic of Java. When you press the button to get a console redirection, a tiny Java Web Start application comes down to your machine and connects back to the server. With it I can do all sorts of cool things like handle BIOS, take console options on my Operating Systems (I use Solaris, OpenSolaris, RHEL and Windows on my machines), and so on... I am also able to monitor event logs and ensure the components are operating as I expect (heck, I can even flash LEDs to say Hi to all of those frozen lab folks who haven't set up their eLOM).
My biggest complaint with eLOM isn't even with eLOM, its with Mac OS X. I have to tell you, I am not happy with their Java support at all. The eLOM console redirection doesn't work (strange Apple-only Java problem). I went to java.sun.com to get a Mac OSX updated JVM...no luck, have to go to Apple. Go to Apple, they are hung up on Java 1.5. I found the Java Platform Standard Edition 6 Press Announcement, its been out since December 2006.
I found a Java Platform Standard Edition 6 Developer Release at Apple but it only runs on Leopard and I haven't upgraded (and I'm not). So, I'm stuck with the Mac...can't use it in our labs to support my systems. I know people love their Macs but some days it reminds me a little too much of OS/2. Great promise with a lack of application support.
I bounced to another platform (with its own Java 1.5 that actually works), the JVMs on my Solaris machines work fine as well as on my Windows laptop and home systems.
In the end...lovin' the eLOM!
Monday Apr 21, 2008
Have you ever been attacked by a plastic grocery bag on the way to work? One of those ones that float through traffic freaking you out? And then it seems to somehow manage to go under your car and pop out the other side only to terrorize yet another driver! Kai and I did some calculations with the bags.
Each bag is a bit over a 15 inch square.
If I didn't use my cloth bags and reuse the plastic "overspill" bags, I would use about 10 per week (why they pack a single bag of potato chips in another plastic bag all by itself is pretty far beyond my comprehension). So, I work my butt off to save at least 5 plastic bags a week (usually I can save all 10). Its probably reasonable that every family could cut about 5 bags off their plastic bag diet if they haven't already.
Saving my 5 bags each week saves about (5 bags * 52 weeks) 260 bags per year. That's saving about (260 bags per year * 15 inches) 3,900 bag inches long (15 bag inches wide) and that is about (3,900 bag inches / 12 inches per foot) 325 feet. Each year saves about the number of bags to go across an American football field and an end zone.
(Yes, that's the high school rugby team playing on the football field...I know they don't need a bunch of plastic grocery bags flying across the field...and they do, quite often).
Let's say all of the families in Highlands Ranch (about 20,000) stopped using about 5 bags a week too. That would be 325 feet * 20,000 people = 6,500,000 feet (15" wide). That's 1,231 miles of bags (15" wide).
The folks of Highlands Ranch could save a strip of plastic bags that runs from Denver to San Francisco (as the Google Map flies).
View Larger Map
Well, that's a lot of bags. Who knows the countless number of car accidents that would be avoided if we all saved a few plastic bags. Reuse the ones you have and get, recycle them, and in so many cases...don't even use them in the first place.
While thinking about the surface area that these bags cover is convenient and somewhat astounding, also keep in mind
- Energy is used to produce these bags
- The bags are comprised of materials like plastics and oils
- The bags take up space in landfills (not much...but it is space)
- The bags won't decompose for 100s or 1000s of years
Next time just think: "Do I really need that bag or can I just carry my sack of potatoes?".
Monday Apr 14, 2008
I had a section to do for Kai to become a Wolf in Cub Scouts...luckily, the pack has decided to go out and do some neighborhood cleanup. Still, I was interested in completing my fact finding mission anyway. My goal was to create a set of information on how "little things" add up to "big things" (remember "think globally, act locally").
There is a lot to choose from in my life, I figured I would take them one by one and see if a little "group think" can help me fix any logic problems. After the Jack Johnson Curious George Soundtrack song, I figured I would categorize into:
And I would take a few simple tasks a child could complete with their parents:
- Encourage your parents to hang out 2 loads of clothes a week
- Stop using 5 plastic bags a week
- Use refillable water bottles instead of purchasing bottles of water
- Walk to school
I haven't dug up all of the facts yet, but I'm committing to it here on the blog!
First task: Encourage your parents to hang out 2 loads of clothes a week.
(Yes, that's one of the loads I hung out today)
I have an LG Tromm Dryer (courtesy of a good quarter or two here at Sun Microsystems). It runs at 240 Volts / 30 Amps, ouch. That's about 7200 Watts per hour. A load of laundry takes about an hour to dry. I've committed to hang out 2 loads per week of laundry, 14,400 Watts per week * 52 weeks = 748,800 Watts saved per year.
I was browsing around the web trying to find the conversion and I found the Ask a Scientist Web Site that said 1gm of coal could power a 60w light bulb for 550 seconds (let's do some rounding and just say 9 minutes). Now we have to do some normalization.
A 60 Watt light bulb with 1 gm / 9 Minutes = 6.67 grams / hour.
My two loads of laundry actually use 14,400 Watts per week (that's two hours of power), for 1600 grams of coal per week * 52 weeks = 83,200 grams which, for my fellow U.S. citizens is 182 pounds.
Another site estimated the creation of 1 Kilowatt for each pound of coal (roughly), which would bring us to about 748.8 pounds instead of 182. The first calculation was in an ideal world, the latter appears to not be in an ideal world.
Doesn't seem like very much does it. I'll use the 182 figure to be conservative. Each of those pounds of coal produces about 3.7 pounds of CO2 so on the conservative chart, I am stopping about 673 pounds of CO2 from going into the atmosphere.
That's quite a bit, but is it worth the trouble?
My little suburb has about 80,000 folks in it, let's say there are 20,000 homes and each one of those families could save 136 pounds of CO2 from going into the atmosphere. That would be about 13,460,000 pounds of CO2 that does not enter our atmosphere each year.
Is that interesting yet? Probably not in the context of all of the CO2 created in a year, but I think that is starting to get interesting.
For a child, what is 13,460,000 pounds?
I saw on Yahoo Answers that a Pontiac Vibe is approximately (and conveniently) 2,700 pounds.
(Image borrowed from Edmunds)
My little suburb of Highlands Ranch, if each family would hang 2 loads of wash out per week rather than dry them, would keep about 4,985 Pontiac Vibes from entering our atmosphere and hanging over us each year. Imagine what a country like the U.S. could do by hanging their clothes out to dry?
If you see any MORE logic errors, please let me know and I'll keep trap and fix them.
Note: I was off by a power of 10 (sloppy night time work) and my dryer was 240 volts. Michael Lyle sent the correction and also noted that I should use an ammeter to get an actual measurement. Stay tuned!
Thursday Mar 27, 2008
My family and I took a brief vacation this weekend and made our way to Washington D.C. for a little R & R. We enjoyed 2 and a half days of sights, tours, history and we even squeezed in a little time for the pool. For those of you that have been to D.C. (or live there), you know that 2 and a half days only allowed us to scratch the surface of the United States cultural base that is alive as well as preserved in the city (and often within a few blocks of the National Mall).
There are so many thought provoking and emotional moments as you move around that after two and a half days I found myself almost completely wrung out. We saved the Congressional Gardens with the Vietnam Memorial, World War II Memorial, Lincoln Memorial, Korean Memorial, and the others for the last day. The artistry and the thought that went into these memorials is astounding and the emotions that they pull out of you put you into knots.
I won't list everything we did on the whole journey over the weekend. For my youngest son, going up the Washington Memorial (and our need to start standing in line for tickets at 6:30am) will probably be the most impacting moments. For Shaun, hopefully the Vietnam Memorial and the Pederson House. For me, who knows, the Bill of Rights, the Constitution, the Declaration of Independence, the Magna Carta...simply amazing, but overall I can't name a single moment that wasn't worth its weight in gold.
Professionally though, the National Archives had to be one of the most thought provoking of our stops.
Here is this large building, with all of these physical manifestations of our history on display and in vaults around the building. The Constitution, the Declaration of Independence, the Bill of Rights, the Emancipation Proclamation and more than I could ever list here. It was "Magna Carta Days" at the National Archives as well and one of the four remaining copies of the 1297 Magna Carta from King Edward I was on display.
Here are a few thoughts that went through my head:
- With infinite, perfect copies of digital content, what makes a digital entity "unique" and "awe inspiring"?
- How is our country going to preserve and revere a digital creation 710 years from now?
- How do you know what digital creations are worth preserving since you can hit a button and destroy them so easily?
The first of these seems entirely out of place with storage technology, but when you stand in front of the Declaration of Independence it makes you wonder what digital content could actually have this impact on a person and how you would embody that digital content. The record companies are struggling with it as well. In addition to digital download content, the companies are trying out releases on USB thumb drives as well as larger packages and "Deluxe" sets. Books are going to struggle through the same revolution (as magazines already have) of being bundled as bits with little or no "branding" or "artistry" about the packaging. How does one recreate that sense of uniqueness when the content is merely a bunch of bits that gets flattened into 10 songs amongst 8,000 on an iPod? Really, what "value" do those songs and books have anymore when they can be passed around at will and are part of a great "torrent" of traffic into and out of our computers? Something really has to "stick" to remain on "top of our stereo" these days.
And as for the Declaration of Independence...what a clean and simple document. The document itself hung in windows and is incredibly faded and worn down. Only after time passed did our country seek to formally preserve it for posterity. Perhaps we caught it in time to save it from deteriorating any more. But one still has to ask, is it the single original document that retains the significance or is it the content that remains significant. If it is the content, we wouldn't store the original in a huge underground vault and protect it as well as Vice President Cheney, would we?
Having seen the original, I would have to argue that there is something incredibly unique about it, it actually holds more reverence (for lack of a better word) than one of the many copies of it. So how does one reproduce that "reverence" in a digital world?
If that is not enough to think about, we have to think about digital preservation. The Magna Carta of 1297 has withstood time for 710 years and is in wonderful shape. What digital storage technology today do we have that can withstand decay for that length of time (of course, one could argue that some rock etchings have withstood time for 1,000s of years). Let's put this in perspective, today's disk drives and SSDs are generally spec'd for 5 years. If I want to preserve my family's pictures for 710 years, I would have to ensure the data was migrated 142 times. Hmmm, I'm not sure if my kids and their kids and their kids are up for that.
It appears that CDs and DVDs may have a lifespan of around 50-200 years if you preserve them properly. That is getting pretty reasonable...of course, they haven't been around for 50 to 200 years so they are certainly not battle tested like carving on a good rock. The National Institute of Standards and Technology appears to be looking heavily into the longevity of optical recording media. DLT appears to have a shelf-life of around 30 years if preserved properly.
Let's say, hypothetically, that you solve the problem of the storage media (perhaps a self migrating technology in a box that guarantees infinite lifespan and that, itself, produces the new disks and technology to ensure fresh DVDs are always built). Now you have two additional challenges (at least):
- Maintaining the integrity of the data (how do I ensure) that the data that is NOW on the DVD is the original data
- Maintaining the ability for outsiders to inspect and recall the stored information
The first of these seems obvious, but is actually quite difficult. Checksums can be overcome with time (imagine compute power in 700 years!) and we can't guarantee that the keepers of the information will not have a vested interest in changing the contents of the information. We see governments attempting to re-write history all the time, don't we?
Let's take a simpler example of what happens when "a" byte disappears. Recall Neil Armstrong's famous quote: "One small step for man ... ". Well, after a lot of CPU cycles and speculation and conspiracy theories, it turns out that we now believe that Neil Armstrong said: "One small step for a man...". It is a fundamentally different statement (though there is no less historical impact). This data is only 40 years old, but consider the angst in trying to prove whether or not "a" was a part of the quote. What happens when a government deliberately alters, say, the digital equivalent of an "original" 2nd Bill of Rights written in 2030?
One more thought for the day, since I really do have to work and if you have made it this far, it is my duty to you to free you of my ramblings.
We know for a fact that the English (dare I say...United States dialect) language is evolving. Even after 200 years there are phrases and semantics and constructs in the Declaration of Independence that require quite a bit of research for the common US citizen. Take the following paragraph:
He is at this time transporting large Armies of foreign Mercenaries to compleat the works of death, desolation and tyranny, already begun with circumstances of Cruelty & perfidy scarcely paralleled in the most barbarous ages, and totally unworthy the Head of a civilized nation.
There is the obvious use of the word perfidy, a word that has since all but disappeared from common speech in the United States.
Looking deeper at the paragraph we see evolution in spelling (compleat). There is also a fascinating use of capitalization throughout the Declaration of Independence. The study and usage of capitalization alone could be worth the creation of long research papers.
What does this tell us? The content and meaning of a work lies often with the context and times in which the work was created. How does one retain this context, language, and ability to read the content over 700 years? This is not a small problem at all. There are entire cultures lost or in the process of being lost as the language and the context is lost, consider the United States own Anasazi culture as an example.
A computer dialect (protocol, standard, information model, etc...) are themselves subject to evolution and are even more fragile than spoken language itself. A change in a capitalization in an XML model may break the ability of pre-existing programs for reading and migrating information, resulting in lost information. Once you break a program from 200 years prior, how much expertise will still exist to maintain and fix that program?
Crazy things to think about. Personally, I believe we are in a fragile place in our history where we could lose decades of historical information as we transition between written works and digital works. As part of my night job I'm trying to get more involved in the Sun Preservation and Archiving Special Interest Group (PASIG) to learn more about what our customers are doing in this area. I'm also trying to reorganize my own home "infrastructure" to be more resilient for the long run to ensure that my family's history does not disappear with my computers.
There are significant challenges in the computer industry all over, but preservation of history is one that our children and our children's children will judge us with. USB thumb drives will come and go, but hopefully our generation's digital treasures will not go to the grave with us.
|
|
|
|
tenks