Georg Edelmann's Weblog Georg's Weblog

This is an interesting article on Samsung's SSD product line. It's refreshing to see how fast innovation happens in this space. It is encouraging to observe, and bodes well for a system provider like Sun. After all, it's when SSD are coupled with servers, storage, operating system and applications that real business problems are being solved.

Enterprise Flash just hit the one million write cycle mark. See this Register article.

Sun Storage 7000 came in third position of eWEEK Senior Writer Chris Preimesberger top 10 storage stories of the year 2008. Check it out at this page.

I've been completely swamped with work at the moment, which meant that blogging fell to the way side for now. I have tons of great idea to talk about, but not enough hours in the day to get them on paper.

The good news is that my being busy is all because our ISVs want to hear what Sun is doing around Flash. I spent quite a bit of my time in ISV briefings, talking about Flash and how Sun's Flash strategy can solve some of the storage performance problems everybody in the universe seems to have nowadays. This is pretty exciting.

For some time, I've been reading Chuck Hollis' blog, and the more I read it the more I enjoy it. Chuck has an interesting style. He's not afraid to rattle some cages. In a world that's becoming ever more increasingly politically correct, people who speak their mind are refreshing. This is not about being right or wrong, and who is to decide on that anyway. It's about taking sides, making a point, speak up on what you belief in. Thanks, Chuck.

Flash memory gets a lot of press nowadays touting it as the cost-effective solution for your storage performance needs. With this blog entry , I'd like to explore some of the rationale behind this enthusiasm. Let's start with some numbers.

According to this IDC whitepaper by 2011, the digital universe will be 10 times the size it was in 2006. Another interesting aspect is that approximately 70% of this digital universe is created by individuals, but enterprises are responsible for the security, privacy, reliability, and compliance of 85%.

With that, I think it's safe to assert this data explosion will cause a need to increase storage performance.

Interestingly, the server world has reacted to the steady increase for compute performance with ever faster CPUs, higher memory capacities, multi-core designs, increased server utilization through virtualization, etc. Despite some innovation, it seems that the technology in storage land lags behind. Hard disk drives, even the fastest 15K RPM can not feed servers fast enough. They are some hundreds of times slower than what today's servers are capable of so they spend most of their time waiting for data after a request. They are starving (see Dave's blog entry here for quantitative example). The traditional remedy of adding more expensive DRAM may no longer suffice as data sets double every couple of years. Today, some applications are being chocked by spinning disk drives which which are causing storage latencies and I/O bottlenecks.

What is the solution here ?

A good initial example is Sun's approach, which is to place Flash-based storage in the form of SSDs to accelerate the overall storage I/O performance of the Sun Storage 7000 family with the use of Hybrid Storage Pools. By strategically putting SSDs in hot areas on your storage subsystem, Hybrid Storage Pools make great use of the performance characteristics of SSDs. On top of that, current SSD capacities are well suited for this approach.

But what if your need both : performance and capacity ? or in other words, what do you do if you need the performance of SSDs and the capacity of HDDs ?

Well, I guess we'll see what innovation will surface in this space. For now, keep watching Sun announcements.

In the run-up to yesterday's Sun Storage 7000 (aka AmberRoad) launch, I had the opportunity to brief a series of our key ISVs on our vision, plans and key features of AmberRoad. I wanted to write up some of the observations, comments and feedback we encountered. There were two distinct categories of ISVs. First, the type of ISVs that adds to the AmberRoad ecosystem as a storage device. Think along the lines of ISVs offering backup software or anti-virus scanning engines. And then there was the category of ISVs who use NAS devices as a data store. The discussion with both type was distinctly different.

Let me start with some common reactions from the ISVs. In no particular order, this is the feedback we received.

Presented with the vision of building storage appliances from industry-standard hardware components, OpenSolaris and open-source software, the most common reactions was that it's about time that all the innovation happening in server land is being transplanted into the storage world by a major IT vendor. The time for closed, proprietary storage devices has come and gone.

All ISVs, and especially the ones that consume a lot of storage themselves, homed in on the possibilities of the analytics capabilities can offer them to figure out performance issues. One ISV commented that their customer's approach to solving performance problem was to buy more storage. Let's agree to call this a sledgehammer approach, not really viable in these tough times. The dtrace-based analytics of AmberRoad opens up the possibility to finally determine the root causes of performance issues with a scalpel.

By far the biggest applause we got from our ISVs was for the decision to not license data services features separately in AmberRoad, and to include all software features with the purchase of the product. Find out what they are by installing the "Sun Unified Storage Simulator". As they say, who reads spec sheets if you can find out for yourself using the real thing. This also means that customers and ISVs don't have the hassle of managing license keys. Neither do we.

More often than not and mostly half way through the briefing, the questions of "can we run our application directly on AmberRoad" came up with predictable reliability. After all, AmberRoad is running OpenSolaris, a general purpose operating system. So, why not run applications on the storage device ? Interesting question. From a technical perspective, this should be possible. However, at this point of the AmberRoad lifecycle, we do not support ISV applications running directly on AmberRoad. We'll see what the future brings here.

We also had some more critical comments, along the lines of Sun's success with the first attempt in the NAS market with the Sun Storage 5000 family. How is Sun Storage 7000 different ? I guess the answer here is that AmberRoad is a vastly different approach. Amber Road is based on industry-standard server technology with a storage personality. Sun can build on decade of experience gained by engineering server hardware and an enterprise class operating system.

Another question that popped up was as to whether we are just copying what Openfiler does. My answer to this question was "Yes" and "No". Yes, in the sense of Openfiler's vision of using a general purpose operating system for a storage device. No, in the sense of that AmberRoad is a fully integrated hardware and software (FISH) stack backed by a major IT vendor.

A more technical questions from the backup vendors was as to whether we support NDMP. The answer is yes. For all software features of AmberRoad, please refer to slide 14 of this presentation.

When the discussion drifted towards performance, the most common reaction was ... dare I say it ... disbelief. How can Sun claim superior performance if AmberRoad is entirely based on commodity parts ? Well, let's leave this discussion for another blog entry. It you can't wait, start exploring the the AmberRoad performance page.

Without exception, ISVs saw the great potential of AmberRoad. All of them were keen to get their hands on a system.

Feel free to leave any comments, especially if you are an ISV and are interested in Sun Storage 7000.

I finally found some time to do a crude performance analysis of my home-grown D2D backup configuration. I started off attaching the two USB disk to a Sun Blade 150. As per a previous comment, with the Sun Blade only support USB1.1 transfer rates, the performance was pretty bad. It took in the order of 15 minutes to backup 1GB of data. With approximately 4Gb worth of pictures and video files to backup, I gave up on this setup.

I attached the two drives to another server, that supports USB2.0. Again, running OpenSolaris, and again the same ZFS configuration as mentioned in my previous blog entry.

To get a feel for the performance, I created a 1Gb file called "bigfile" on another Solaris box on my network. Then I copy it over to my backup server with as follows :

$ time scp bigfile georg:/backup/ge29057

real    3m9.760s
user    2m0.063s
sys     1m3.858s

So, 3 minutes to backup 1Gb of data. That's approx 12 minutes for a complete backup of all my pictures and video files. That's good enough for me.

Today, my two Cavalry 1TB USB 2.0 3.5" External hard Drives arrived representing a total investment of 2 x $119.99 purely for backing up my collection on pictures and video. I timed myself on how long it took me to have the drives operational. Here is the run down :

• Step 1: Open the box and unpack : 2 minutes.
• Step 2: plug in the two drives to power, and connect them to the USB port on my server : 1 minutes.
• Step 3: configure the drives as a ZFS mirrors : 5 minutes.
• Step 4: created the Samba share : 1 minute.
• Total : 9 minutes.

Here are the commands, I used :

# uname -a
SunOS piaffe 5.11 snv_100 sun4u sparc SUNW,Sun-Blade-100
# zpool create backup mirror c1t0d0 c2t0d0
... edit smb.conf to add a share ...
# svcadm restart samba
# zpool list   
NAME     SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
backup   928G  91.4M   928G     0%  ONLINE  -
...

I'm impressed. This was very, very easy. I also played around with turning one of the drives off while writing to the ZFS filesystem to see what happens. As expected, ZFS took care of everything. It could not have been any simpler.

As a next step, I'm going look into the kind of performance I can get with this setup.

I've been pondering on how I can best backup my ever growing amount of digital pictures and home videos for some time now. Here's where I am as today. My selection criteria are :

• It needs to be cheap. No, not cost effective, I mean dead cheap.
• It has to be quite. My wife HATES noisy computer gadgets.
• Did I mention, it has to be cheap ?!?!
• I want to be able to do backups over the network.
• It has to work with OpenSolaris and ZFS.
• It needs to reliable. After all our families histories is on those pictures and videos.
• I need to be able to backup OpenSolaris and Microfot Windows clients.

I already have an OpenSolaris system as a file server in my basement. At the moment, I am tempted to buy two 1TB USB drives, e.g. this type, mirror them with ZFS and connect them to my file server. I could then install Amanda on the server, and use the system as my central backup server. Anybody can see any flaws with this plan ?

I got forwarded two more interesting articles on SSD in the Enterprise from Storage Networking World. The Register article here talks about how all major storage vendors now have a SSD strategy, except for 3par.

What really caught my eye was the following paragraph from the Burton group blog:

"...Here's what the market will really love: a blended system with SSDs for performance and terabyte SATA disks for capacity. To make this work, auto-tiering will be needed under the covers, transparent to users. Ideally, this product would allow policy-based data movement leveraging usage patterns and storage costs. [...] SUN's got some interesting ideas using ZFS...."

In my humble opinion, the writer gets the point completely. The vendor that has the technology to seamlessly integrate SSDs in the data path has a compelling story to tell. As for what exactly those interesting ideas from Sun look like, we shall see. Very soon.

... or better SSD for the Enterprise are here.

Just in case you missed it, Sun has a stated strategy to incorporate SSDs into its product strategy, see for more details.. Reading this article, as well as this one, it becomes clear that Intel is clearly on the same bandwagon. My team is currently working on some ISV applications to figure out how real life applications can make use of the IO boost from SSDs. Watch this space for more details.

If you ever doubted that Sun's engineers are focused on performance, check out this new page. In short, Sun is a performance driven company. I'd know, the team I am part of lives and breathes product performance 24x7.

I'm still a bit jet lagged from my recent trip to Germany, so over the last couple of days I got up earlier than usual. Today it was at 5am. I actually don't mind, as this gives me some time to catch up with things.

When I went to my office a couple of minutes ago, I quickly noticed that my DSL was down. I call my provider's support line, which I am on hold for the "next available analyst" at the moment. The plan is simple, I'll ask the support person to throw away the script and reset my DSL port as politely as possible. I've been here before. It seems to me that any connectivity problem results in a port reset anyway, so why not start here. Anyway, that's a topic for another blog.

While on hold, the system announced that the provider experiences network problems in my area. So, I guess I am stuck without Internet access for a bit. I'll keep holding to ask for an ETF (Estimated Time to Fix). In the meantime, while still on hold music, my mind started to wonder. I started my professional career before everybody was connected. What in the world did we do with our computers before we had the Internet and the World Wide Web ?

Hmm, I remember finding computers to very useful for the support job I was doing at the time. We used an Oracle-based support management system at the time. Calls came in via phone or fax. I had several dedicated X25 lines, allowing me to telnet into my customers' servers to diagnose issues. Some of my customer were security sensitive agencies. The only way to communicate was via faxes, as they needed to be able to keep a trail of all communications.

Our knowledge database was a home-grown, file-based system. It used grep(1) to find articles, and it was able to extract fax numbers for customers from our support database, and then fax support articles to customers. We regularly created CDs with all the patches and articles to send to our supported customers. As a support engineer, I was goaled to create new articles. Over time, our support team created quite a comprehensive knowledge base. I recall that we could resolve up to 95% of all support calls with the data we had in our knowledge base.

I also remember the beginning of Email, and reading Email via mail(1). Most, if not all, Email was internal to our company. SPAM was not invented yet. 10 Emails a day was considered a heavy Email load.

Later in the job, more and more customer got X25 connectivity. We used these connections mainly for UUCP-based Email, and to FTP updates to our knowledge base to our customers , and to exchange Unix core dumps and other diagnostic logs. It did not really change the way the support department operated.

Then, and I vividly remember this part, somebody talked about the new thing called the World Wide Web. I was sent a tool called "Lynx", which was referred to as "WWW viewer" by my colleagues (Note: most of us where using Wyse terminals at the time). We installed Lynx on our server, and started to play. It was pretty useless. There were a couple of WWW sites, mostly research stuff. Granted, it was great to link content together, but finding pages with valuable content for my work was a non starter. After we played around for a bit, we decided that this WWW thing is never going to take off. We must have felt the same way as one of the first telephone users.

After a brief period, somebody came up with the idea of making our support knowledge base available over the WWW. We could reduce the number of CDs we had to manufacture and send out to our customer, and we could link articles together. We pinged our customers to see if this approach would fly. Most of them were positive, but insisted to still get the maintenance CDs on a regular basis. We went ahead and uploaded our knowledge base to the WWW. Over time, fewer and fewer customer wanted the maintenance CDs. If memory serves me right, I think it was my government agency that was the last customer to request our support CDs, as they were not allowed public Internet access.

Fast forwarding to today, I can't think on anything I do that is not based upon some kind of Internet service. Taking my DSL line out for a couple of hours is a painful reminder of that. I guess "The Network is the computer" also means that without network your computer is no great use.

I'm just done listening to a presentation on MySQL from my esteemed MySQL colleague Robin Schumacher. Robin is director of product management at Sun/MySQL. Not being a database expert, I thoroughly enjoyed Robin's whirlwind tour of MySQL. I want to note of a couple of highlights that caught my attention :

• MySQL Community Edition and MySQL Enterprise Edition are feature identical. Wow, that was a suprise to me. I hate it when the Community Edition is distributed as "crippleware".
• Kickfire offers a SQL chip. If I understood correctly, this is a piece of silicon that speeds up your SQL statements. Wow again. I thought the time for custom built silicon for a specific purpose came and went, and the market has long ago decided to pack the intelligence into software, and to use commodity processor units as base. I guess I was wrong here.
• Lastly, I really liked Robin's MySQL value proposition. To quote Robin : "...MySQL offers 80-90% of the functionality of expensive proprietary databases at 10-15% of the cost...". As value statements go, this is as straight forward as it gets. Wow, for the third time.