Management Monogatari: Tales of Resource Management from Mark A. Carlson

The SNIA has released early drafts of a couple of documents created in the new Cloud Storage TWG. The Cloud Storage Reference Model sets out a model of cloud storage elements that describes a logical view of their functions and capabilities using a descriptive taxonomy. The purpose of the model is to form a basis upon which industry efforts can be organized, needed standards identified and vendor products can be described by vendor independent terminology. In addition, the model is used to describe standard interfaces for cloud storage.

The SNIA will create a new interface called the Cloud Data Management Interface (CDMI) that will serve as both a functional interface to store data in a cloud, and as a management interface for the data that is stored there.

SNIA is soliciting feedback on the model and use cases in order to shape this interface work. If you would like to get involved, there is a Google Group you can join.

DMTF announced their Cloud Incubator standards group and Sun is among the companies on the leadership board. This work will complement and be coordinated with the work already going on in the Open Grid Forum and SNIA.

In the cloud computing space, the key technology driving the Infrastructure as a Service (IaaS) offerings (including Sun's) is system virtualization. DMTF has been working on system virtualization standards such as the Open Virtualization Format (OVF) specification and the System Virtualization Profile, which focuses on virtualization aspects related to host systems and their resources, such as modeling the relationships between host resources and virtual re-sources. Further it addresses virtualization-specific tasks such as the creation or modification of virtual systems and their configurations.

The DMTF has actually developed multiple profiles related to virtualization and leveraged others that had previously been developed. In addition to the technical work, DMTF has an initiative to promote their virtualization standards called VMAN.

Quoting from the VMAN website: "The Virtualization Management Initiative (VMAN) from DMTF unleashes the power of virtualization by delivering broadly supported interoperability and portability standards to virtual computing environments. VMAN provides IT managers the freedom to deploy pre-installed, pre-configured solutions across heterogeneous computing networks and to manage those applications through their entire lifecycle. Management software vendors will offer a broad selection of tools that support the industry standard specifications that are a part of VMAN, thus lowering support and training costs for IT managers. Thanks to the Open Virtualization Format (OVF) standard within VMAN, ISVs can create a single pre-packaged virtual appliance that can run on customers' virtualization platforms of choice. Tools based on the VMAN Profiles enable consistent management and monitoring of these virtual applications across the virtualized platform. These technologies will allow ISVs and platform vendors to focus their development resources on higher value features of their products instead of needing to create different versions of products for each environment." Sun just announced that it's VirtualBox x86 virtualization product supports the OVF standard. So as virtualization technology becomes ubiquitous, being deployed on desktops, enterprise data centers and now public clouds, the need for standards becomes even more important. DMTF’s Open Cloud Standards Incubator will focus on addressing these issues by developing cloud resource management protocols, packaging formats and security mechanisms to facilitate interoperability. You can read the charter of the group to learn more.

SNIA announced the formation of the new Cloud Storage Technical Work Group. SNIA has a long history of standardizing interoperable interfaces for storage networking, and cloud storage is really just the next step in networked storage.

There are several example use cases for cloud storage, including:

Store my file and give me back a URL (i.e. Amazon S3)

Best Effort Quality of Service?

Provision a filesystem and mount it (i.e. WebDAV)

Quality of Service specification via provisioning interface

Give me Filesystems/LUNs for my Cloud Computing

NAS box in the cloud…

Store my backup files until I need them back

Maybe offer me a local cache as well

Archive my files in the Cloud for Preservation/Compliance

Maybe offer me eDiscovery services, “tape in the mail” retrieval

Store all my files, allowing me to set the Data Requirements, let me cache and distribute geographically

Policy driven Data Services based on Data System Metadata markings

One of the immediate needs that the TWG hopes to address is a reference model and some common terminology just to sort out all the uses of cloud storage. This work is targeted to be available in draft form this summer. Based on that common understanding, the group may also start to develop standard interfaces, but will not likely get out in front of actual deployments. The best standards are driven by requirements derived from actual customer pain points. Only when multiple vendors have a common set of capabilities, does it make sense to standardize the interoperability between them. Sun is one of the foundational members of this new TWG, and I am looking forward to some exciting technical activities in the weeks and months ahead.

USENIX puts on a storage conference every year called FAST: File and Storage Technologies. Sun is a Platinum sponsor this year and is holding its semi-annual OpenSolaris Storage Summit on Monday. The summit is free and is being held at the same hotel (San Francisco Grand Hyatt) as the FAST conference. There is a wiki that you can edit to register for the summit, and use to find out more about the event. The FAST conference itself is a great place to find out the latest research from the academic side of the storage industry. I plan to attend the technical sessions. One presentation in particular I am interested in is on Thursday. Brandon Salmon is talking about a system he helped design that manages all the data on your devices (laptops, desktops, PDAs, Phones) and controls the redundancy and location of that data based on how much you value it (or fear it's loss). I met Brandon at the annual Parallel Data Lab (PDL) retreat put on by CMU. Sun is also a sponsor of PDL. I will be involved in a couple of Birds of Feather session during FAST. On Wednesday night, you can find me at the Open Storage BOF, where we will discuss the use of open source software to create storage solutions. Thursday night I'll be leading a BOF on Cloud Storage, a trend that seems to be getting some traction lately. There is some interest in creating standard APIs for this type of storage, while others feel that standards at this early stage might stifle innovation. Should be an interesting discussion. Hope to see you there.

The blog-o-sphere has been abuzz of late with postings claiming that SMI-S is dead, or dying. The claim being that the standard API/protocol has not fully replaced the need for proprietary ones. As if any management standard ever did. Perhaps if we all stopped adding features to our products, the standard could finally catch up ;-) You really can't compare management standards to to protocols such as TCP/IP and HTTP, where there is a strong bias *against* adding features (witness IPv6).

All these issues are hot topics within the standards organizations such as the SNIA and DMTF as well. Eight years later, the scope of resources that are able to be managed via a single protocol and common model has grown dramatically. The set of tools, mostly open source, to create solutions in this space is very impressive. What puzzles me is why anyone would re-invent all this infrastructure from scratch?

Mostly, the use of proprietary protocols and APIs is historic. They came first. SMI-S, like any other "check box" feature gets thought about later in the development. Because the vendors don't see it as "strategic", they only implement the minimum to pass the certification tests.

Once this gets established in a product line, it's difficult to switch things around, but there really is no excuse anymore to start a new product line with a proprietary API as the "primary" means of managing a resource. The standards can easily be extended to cover the whole functionality of any given product. By leveraging SNIA and DMTF profiles and protocols as the primary interface, the implementation becomes rich and robust because there is no other interface.

Some vendors do this today, but you'd never know it. What needs to happen is to have this become a differentiator between products, and get customers to ask for vendors to provide the full functionality through the standard-based interface. We really don't need to standardize any more features in order for this to take place. It's more of a messaging problem than a technical one.

SMI-S may never be "Done" as long as we keep innovating in the storage space, and other implementations drive a need for a common way to manage those features, but who cares? It's certainly complete enough to be the primary interface into your next storage purchase, if you call the vendor on it.

A this year's Storage Developer Conference, I gave a talk about a new model that the SNIA Technical Council has been working on. It's called the Storage Industry Resource Domain Model and we are using it to align various Technical Work Group (TWG) charters and to provide a baseline for new terminology in this space. We expect the industry to use it to map their products onto the model and talk in industry standard terms about what their products do. Here is the video: You might also want to download the iPod version for offline viewing. The slides can be downloaded from here

SNIA has recently blessed the 1.0 version of the eXtensible Access Method (XAM) standard. The specification will soon be sent to ANSI and ISO for national and international standardization, but you can get started on coding to XAM now.

SNIA has also released the first working draft version of the XAM SDK. Download the source code from here (BSD License). This includes the source for a complete C XAM Library, a complete Java XAM Library and an early version of the Reference VIM that runs on any Java platform using a plain file system as the backing store.

If you are new to XAM and want to learn more, there is a unique opportunity to get educated at the upcoming Storage Developers Conference in late September.

There will be a XAM Tutorial on Monday, XAM Hands on Lab for Developers (HOLD) sessions on Monday and Tuesday and more XAM related session talks on Tuesday as well. SDC goes all week until Thursday with even more content on other storage technologies you won't want to miss.

The XAM Initiative is planning to fund a limited number of scholarships to SDC for XAM Developers that otherwise would not be able to afford the registration fee (as much as $1395). If you are interested in the scholarship, you will need to fill out this online form to apply for the scholarship. We will be evaluating the entries based on XAM usage and need. Once you get your hands on the XAM SDK and have your hands on experience at SDC, be sure and sign up for the XAM Developers Group where you can ask questions of the experts and get help with your development. The experts and other developers there can give you a hand.

The date for the OpenSolaris Storage Summit has been set. We will be hosting the event at the Santa Clara Hyatt Regency hotel on the 21st of September, 2008. This is right before this year's Storage Developer Conference at which Sun is a Platinum sponsor.

We already have a couple of keynote speakers lined up: Ben Rockwood and Mike Shapiro. It will take place all day Sunday, so if you are coming to SDC this year, come a day early and participate. We will have the first ever OpenSolaris Storage Community meeting (face to face). A great way to meet some folks that you may only know by their email addresses. Everybody has an opportunity to give a Lightning talk and/or put a poster together about what they are working on or doing with OpenSolaris storage.

Registration is via a wiki page here: http://www.genunix.org/wiki/index.php/OpenSolaris_Developer_Summit_08. Just add your name and contact information into the Attendance List table and you are registered! Attendance is free and we hope many folks from the community will attend this year. Also, you should add yourself to the summit list, even if you are not sure you will be able to go, so you can keep up to date with the planned activities. We may create ways to participate even if you can't make it to Santa Clara this year (if we get people asking for access). Once registered, scroll down to the bottom of the wiki and suggest some topics or volunteer to participate as a poster or lightning talk. Fill in some details about yourself (or someone else) at the bottom of the page.

Check it out, get involved, and I hope to see you there! It will be a blast. More details later...

Sun has released the source for the Fibre Channel over Ethernet (FCoE) target driver on OpenSolaris. Note that this is a complete software implementation of FCoE - no special HBA is required. It will operate with a simple Ethernet NIC as hardware (e1000g/bge). It will interoperate with the Linux initiator from the Open-FCoE project.

Modern processors (with multiple cores and threads) have plenty of cycles to do software initiators and targets. Software initiators and targets for iSCSI are pretty much the norm (TCP offload never took off). In the case of FCoE, does it make economic sense to offload the Fibre Channel stack to an offload engine any more than the TCP/IP stack? But that's what many of the existing Fibre Channel HBA vendors are hoping for with FCoE - to sell you an offload stack and charge more for the NIC.

It's too early to tell how this new protocol will shake out in the products that survive in the market, but you can get started now with this newly released code. Download it, play with it, do some performance measurements and get involved in the OpenSolaris community.