Notes from the Main Sequence: Eric Reid - Sun ISV Engineering

"Canary In A Coalmine"
The Police
from the album "Zenyatta Mondatta", 1980

Background

A little project I'd had in the back of my mind for some time was to replace the family fileserver over the Holidays. I'd been using a Linksys NSLU2 for several years, but we were about to outgrow our 500GB capacity, and response time was never all that great. I'd installed Unslung early on, which gave me SMB, NFS, FTP, etc., but the poor little Intel XScale chip inside struggled just to drive one server, let alone several simultaneous access methods.

The NSLU2 was most recently running with two 500GB USB 2.0 drives attached. The only redundancy provided is the ability to backup from the disk to an identical disk once a day - not very robust. I'd been thinking about it ever since I saw Zhong's really good writeup of creating an OpenSolaris fileserver about a year ago. The factors driving my decision were as follows:

1. Green: be as energy-efficient as possible; reuse existing pieces if possible
2. Robust and Redundant: Provide more storage and more robust storage (RAID Z or better)
3. Compatible: Provide the SMB/CIFS (for family) and NFS (for me) access we were accustomed to at home
4. Familiar: Leverage technologies I know:

• Solaris or OpenSolaris
• ZFS
• Shuttle SFF Barebones systems - I'd already built several over these over the years for myself, family and friends

Well, when I saw the rollout for Shuttle's X27D dual-core Atom-based SFF, I knew that was what I wanted. I waited, I planned. Last month the stars aligned, the prices came down, and I acted.

Old Setup 

	Make/Model	Best Current Price	Specs
Server	Linksys NSLU2	US$43 refurbished; US$86 new	USB2 disk I/F; 100Mbps net I/F; running Unslung 6.8
Storage	2 x Simpletech SimpleDrive External Disk drives	US$88 refurbished; US$51 used (EBay)	3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

• 490GB formatted storage
• Nightly automatic backup onto clone disk
• SMB/CIFS access via LAN
• Boots in around 15 minutes
  
• NFS access via LAN (only via reflashing with Unslung)
• ~4.5MB/s write (NFS), ~3MB/s write (SMB)
• ~5MB/s read (NFS), ~4MB/s read (SMB), ~150KB/s (scp, sftp or rsync) - note: any encrypted transfers will be CPU-limited by that poor XScale processor)
  

New Setup

I decided to reuse my existing external disk drives, as well as an existing laptop drive. This meant I only needed to purchase the X27D itself, plus memory. No DVD/CD - I use the network and bootable USB devices instead. To allow for larger, more robust, and more robust storage, I was able to find a used Simpletech drive (same access times, same total raw storage capacity) on EBay.

	Make/Model	Price Paid	Specs
Server	Shuttle X27D SFF	US$139 refurbished	Builtin dual-core Atom 330 CPU @ 1.6GHz; GigE networking
		US$33 new	2GB DDR2 memory
		Existing disk	2.5" 120GB SATA disk
Storage	3 x Simpletech SimpleDrive External Disk drives	Two existing USB disks plus a third from EBay - US$51	3.5" form; USB2 disk I/F; 524GB raw storage each

This setup provides:

• 957GB formatted storage via 3-disk RAID-Z pool
  
• SMB/CIFS, SSH, NFS protocols supported
  
• Boots in around 75 seconds
• Server consumes 26W idle, 32W under load; disks consume power only when accessed, and spin down after 20 minutes
  
• ~14MB/s write (NFS)
• ~18MB/s read (NFS)
• Same disk latency (due to disk spin-up) as original solution
• More onboard processing 'headroom' for future services
  

Configuring OpenSolaris and ZFS

I downloaded the USB Boot Image of Build 129 of OpenSolaris from here, placing the image on a 1GB USB Flash drive using the usbcopy command. After assembling the server, I set the BIOS to boot from USB first, then booted from the 'Live USB'. Installation just worked, and everything seems to be properly supported (haven't tried audio, as this is a headless system).

I'd rsync'ed the contents of the old fileserver to another server the day before (slooowwwww from an NSLU2 - when one of your systems is CPU-limited, make sure you rsync with the lightest-weight encryption possible [I used arcfour]). Setting up the powerful, performant, redundant ZFS filesystem required only a few steps:

# zpool create fs raidz c4t0d0 c6t0d0 c7t0d0   # RAIDZ pool (2+1)
# zfs set sharenfs=on fs   # NFS share                            
# zfs set sharesmb=on fs   # Samba share                         

Still to do:

• Investigate adding cache to the zpool via part of the internal SATA disk (or possibly eventually an SSD, when prices come down)
• More rigorous and apples-to-apples performance comparisions
• Optimizing the OpenSolaris settings and services
  

At Bay Area DrupalCamp today, Sun is handing out USB Flash sticks that boot any USB-bootable X86/X64 box into OpenSolaris with a running Drupal 6 instance. For those of you not at BADCamp this weekend, we've also made the USB image available for download here. It's a little less than 1GB, and fits on 1GB flash drives.

To burn the image to your own USB flash drive:

• From a Windows box: see here
  
• From an OpenSolaris box: install and use /usr/bin/usbcopy from SUNWdistro-const package

For those who'd rather try this from a "Live DVD", the .iso image is here.

Update 19 October: thanks to alert readers who spotted my ill-formed DVD link above. It's fixed now.

A quick note that our friends at Sun BigAdmin have published a short paper of mine called "Joomla! Deployment Guide for Sun Storage 7000 Unified Storage Systems". We wanted to give a more hands-on constructive view of these very cool products from the sysadmin and open-source perspectives. These boxes work, work well, work fast, and bring intelligent network-based storage roaring into the Enterprise.

What to do during those... erm... 'uncertain' times? Why, branch out of course... and that's what lead this intrepid blogger to consider questions like "what's fun?", "what haven't I done in a while?" and "what might be relevant to our imminent New Overlords?".

Relax, Sun Blog Police, that's all Imma say about Snorcle, OK?

I picked 'Oracle Enterprise Linux Certification'. Just cuz. Just cuz I enjoy Linux almost as much as I do OpenSolaris. Just cuz I went and gots me a Red Hat Certified Technician certificate a few years back, also "Just Cuz."  And just cuz it might even be applicable in future endeavors.

For those of you whose knowledge of OEL (sometimes referred to by the overarching 'Unbreakable Linux' moniker) rivals that of Solaris -- Oracle takes the Red Hat code base, strips out any Red Hat identity, slaps on Oracle veneers, and supports the snot out of it. That allows Mssr. Ellison to raise the hackles of yet another rival. Which is good. I'm thinkin'...

Anywho, OEL 5 is RHEL 5 with a different shade o' red paint. If you know RHEL, you know OEL.

An OEL Administration Associate certificate requires the passing of two two-hour multiple-choice exams. If you have a comparable certifications already, you might be given a pass on the first exam - indeed, my RHCT certificate allowed me to test out of the first exam. The second, a more in-depth sysadmin exam, is 113 questions over two hours. Not hands-on, as Red Hat exams are, but also with a lot of 'choose all th apply' questions. Far from trivial stuff, but quite doable if you prepare adequately.

So, as of today, I add the OEL Admin Associate designation to my list of... umm... things. Right along side the RHCT and Solaris and Java Certifications. And my collection of Wired Magazines -- can't forget those!

Cuz, you know, this just seemed like the Thing To Do...

Though documented elsewhere, it's a good idea to propagate for the benefit of the next person.

Problem: Installation of Joomla! 1.5.x on recent versions of OpenSolaris (2009.06 and later) using MySQL fail with errors after install

Symptom: After otherwise successful install of config on MySQL on OpenSolaris, the home page comes up in the browser with numerous MySQL errors, and the MySQL server restarts:

   Error loading Modules:MySQL server has gone away SQL=SELECT id, title, module, position, 
   content, showtitle, control, params FROM jos_modules AS m LEFT JOIN jos_modules_menu 
   AS mm ON mm.moduleid = m.id WHERE m.published = 1 AND m.access =  0 AND m.client_id = 0 
   AND ( mm.menuid = 1 OR mm.menuid = 0 ) ORDER BY position, ordering

Diagnosis: known issue with MySQL 32-bit server as delivered from pkg.opensolaris.org

Workaround: use the 64-bit version of the MySQL server (/usr/mysql/<version>/bin/64/mysqld_safe, et al)

As part of the Sun Open Archive for Digital Repositories effort, Sun has been working closely with Fedora Commons to develop and characterize performance of Fedora on Sun (OpenSolaris, ZFS, Sun Storage 7000 appliances, Sun Storage JBOD storage).

The first phase of performance testing and configuration validation was completed recently, with encouraging performance observed from Fedora, running on OpenSolaris 2008.11 and using RAID0 ZFS Pools. Much, much more testing is needed (and is underway at this writing), but much of what we've seen so far bests any Fedora performance testing done to date.

There's quite a compelling case to be made with the combination of Sun hardware and open source OS and application stack, so we're pretty excited about this work.

Check out the attached documents:

• Poster presented by Sun and Fedora Commons at Open Repositories '09
• FInal version of this first performance testing document released this month - ingest and access performance of Fedora on OpenSolaris, ZFS and JBODs
  

I like VirtualBox. It does what I need -- often times it does many different things via multiple clients on the same server.

Recently, I've been creating OpenSolaris VirtualBox clients, and, again, they do what I need. Except when it comes to accessing OpenSolaris repositories. With default NAT networking, an OpenSolaris client tends to have slower networking, and at times when I'm installing several packages from pkg.opensolaris.org,  I'll get:

 # pkg install amp
          :
          :
 pkg: Maximum number of network retries exceeded during download. Details follow:
          :
          :

Frustrating, to be sure, but there is a way to put up with all the waiting:

  # export PKG_CLIENT_TIMEOUT=300   # Seconds (default=30)
  # export PKG_TIMEOUT_MAX=10       # Attempts (default=4)

Now, although your pkg commands won't be any faster without tweaking the VirtualBox networking stack, they will eventually finish. This works outside of VirtualBox as well.

One of the more exciting and interesting projects I've been involved with this year centers around Sun's Open Archive systems for Digital Repositories - the use of Open Source software and Sun off-the-shelf hardware to implement world-class data preservation/archival systems. For those of you who remember Sun's Honeycomb archival hardware, this new architecture represents a truly open alternative approach - based on Fedora Commons' offering, OpenSolaris, Sun's X64 servers, JBOD disk arrays and (soon) Unified Storage Systems.

One of the first efforts Sun ISV Engineering contributed to involved sizing and performance testing of Fedora atop OpenSolaris, ZFS and JBODs. The results are both encouraging and impressive, and provide myriad follow-on testing opportunities. I've attached the first draft of test report here.

Just what *is* the deal with the wireless coverage @ Moscone Center? My Tecra M2 with brand new OpenSolaris 2009.06 kept the up-down-up-down-up-down all day... never a consistent connection. Anyone have any tips or clues?