All about Grid and Cloud Computing: Discover the limits

 

Fresh off the press: http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=238992

It doesn't mater which programming language you are using Java, Python, or something else. Writing cool kick-ass Unittests is a must. So here are some thoughts and Ideas:

• Design first & design to test. But do not add extra methods juts for testing!
• Always write your unittest first! It will start small (jsut checking the public methods and eventually grow with your code)
• Design and spend time writing test environments with Mocks etc.
• Each method should be at least tested by 3 test methods in your unittest:
  • test for success - Test if the method behaves correctly when given parameters in range
  • test for failure - Test if the method behaves correctly when given invalid parameters, wrong environment etc.
  • test for sanity - Test with a set of input parameters and compare results with expected output parameters.

Next to this I would encourage all developers to use Agile development methodologies like Scrum or similar. And also: please tweet and blog to give your users an insight of what is going on with your product/software.

This blog post is also available here

OCCI has been around for a while now, and for those just starting to fiddle with it here is a 'introduction'. OCCI is more then just a document and specification...'

The OGF Open Cloud Computing Interface Working Group (OCCI-wg) is developing a clean, open API for 'Infrastructure as a Service' (IaaS) based Clouds. IaaS is one of three primary services: Infrastructure, Software, and Platform of the emerging Cloud industry.

OCCI–wg is a working group of the OGF established in March 2009. The group has active membership of over 200 individuals and is led by four chairs from industry, academia, service providers and end-users. Several members are from commercial service providers that are committed to implementing the OGF-OCCI specification.

The idea of cost reduction, GreenIT and on-demand resource planning lead to the idea of having elastic environments which scale on demand. Today's demand for elastic environments in which public Clouds (like Amazon's EC2 offerings) and private Clouds (locally installed resources) interoperate (Hybrid-,Federate- or InterClouds) show the need for standardization. Differentiation in the offerings is done by extensions and prices not by core functionalities. The need to interface all Cloud providers (private as well as public) in the same manner shows the need for standards for core features. Therefore main drivers behind the specification are to ensure Portability, Interoperability and Integration between Cloud service providers and end-users.

OCCI will be a very slim RESTful based API which can be easily extended. Without the overhead of many similar protocols the REST approach allows users to easily access their services. Every resource is uniquely addressed using a Uniform Resource Identifier (URI). Based on a set of operations – create, retrieve, update and delete (CRUD) – resources can be managed. Currently three types of resources are considered: storage, network and compute resources. Those resources can be linked together to form a virtual machine with assigned attributes. For example, it is possible to provision a machine which has 2GB of RAM, one hard disk and one network interface.

The Specification itself is setup to be modular. Therefore it is delivered as a document series. Each of the documents focuses on a certain topic and can be independently used from the others. Until now the OCCI specification consist of four modules: The Core specification, a description of how IaaS based resources should look like, a HTTP rendering for easy machine-to-machine management and a XHTML rendering for creation of human-to-machine interfaces.

The following diagram shows an example RESTful query towards an OCCI compliant service provider:

An HTTP GET operation is performed on an unique URI representing a virtual machine of Solaris zone. The URI encapsulates the instance and provider id. The OCCI specification describes how operations can be performed upon resources (Compute, Storage and Network). Resources can be linked so they described e.g. a Virtual Machine. Each resource type has a certain set of attributes (e.g. CPU speed, Storage size and IP addresses).

The CRUD operations can be performed upon the resources. During the past months two reference implementations have emerged. Based on drafts of the specification a group of the University of Madrid (UCM - OpenNebula) and a group inside of the National Institute of Nuclear Physics (INFN) created reference implementations.

The idea which led to the creation of the OCCI group has been triggered and sponsored by the EU project RESERVOIR. RESERVOIR tries to establish a tool suite for Cloud computing middleware in which the interfaces should be standardized. One of these interface will be an implementation of the OCCI specification. The work done within the group has gained an significant impact in the Cloud community. Blogs, technology related websites, twitter (Tag #OCCI) and journals have driven and reported about OCCI thanks to the open process of the group.

Currently the OCCI group is finalizing and documenting the initial draft specification. After only half a year the OCCI group succeeded in delivering on of the first standards for the Cloud community. This could be achieved thanks to the big momentum the group has had. Due to the specification's high extensibility the group will start to work on extensions after the first draft has been delivered. In future OCCI might become a provider of a complete Cloud computing interface stack which covers IaaS as well as PaaS based offerings.

OCCI plays a major role in today's Cloud computing standardization efforts. The group is actively collaborating with other groups from SNIA for storage and DMTF for management standards. A white paper has been published describing how SNIA's Cloud Data Management Interface and OCCI can work in conjunction. The work is also featured on the http://www.cloud- standards.org wiki where Cloud related standards are coordinated by the major Standards Development Organizations (SDOs).

For more information visit http://www.occi-wg.org/

The guys from OpenNebula made a great announcement on the OCCI mailing list: http://www.ogf.org/pipermail/occi-wg/2009-November/001505.html

Dear OCCI members,

The OpenNebula team is proud to announce the dawn of the OpenNebula Cloud. Although this can be shortened to ONE Cloud, it is in fact two, although both of them are accesible using two interfaces: OCCI and EC2.

* Dummy cloud. This cloud offers an interface to an OpenNebual instance configured using dummy drivers. This means that it will offer a seemingly infinite capacity to run VMs, but it actually won't ever run any VM instance. This 'dummy' cloud is offered to test the OCCI and EC2 interfaces

* Real cloud. The OpenNebula instance that supports this clouds has access to physical server, and will offer the possibility of configure virtual networks, launching real VMs and access them using public IPs. This 'real' cloud VMs will have a limited capacity and is not meant to provide VMs on demand for personal uses, but rather to test OpenNebula cloud functionality.

The aim of these clouds is to allow for interface testing and foster the creation of an ecosystem built on top of OpenNebula clouds.

More information on configuration of the clients and usage of the two clouds can be found here [1].

Best regards,

The OpenNebula Team

[1] http://opennebula.org/doku.php?id=cloud

Cloud Computing is a hype topic, most of you know that. Still there is a lot of interesting stuff going on during the past months and weeks. As one of the chairs of the OCCI (http://www.occi-wg.org) working group I had the great opportunity to present the status and some cloud related work during the Cloud Computing and its Applications conference. I was invited by Ian Foster to present the following slides:

What it comes done to is the following: During OGF27 (http://www.ogf.org/OGF27) we stated to present one of the first standardized Cloud interfaces. We are almost there and soon the specification will be out in the public comments phase of the OGF editor pipeline. More important than having one standard is to have the standards collaborate while each focus on a different aspect. For example the Cloud Data Management Interface driven by SNIA (http://www.snia.org). Now we need some more efforts like demos demonstrating interoperable and portable cloud solutions.

If you wanna know what is going on for OCCI right now: We had a lot of blog posts, mails, etc going on...

BTW the OCCI sessions during OGF27 itself went pretty well and both OCCI and myself have been in the closing remarks from Craig Lee:

I really like OpenSolaris. If you haven't tried it yet you should give it a try. Still there are some things where I choose other operating system over OpenSolaris. Missing ports for some of the application I use is one point... But never the less OpenSolaris has some killer features which makes is worth a try. For example ZFS, Zones or DTrace. One tool I absolutely like when using Linux is the packaging system and the ability to clean up orphaned (e.g. with help of deborphan) packages. After some googleing it seemed that there is nothing in OpenSolaris yet to do so. But here are some tips and tricks to keep your installation clean as long as the feature is not available...

1. On a laptop use ZFS compression: zfs set compression=on rpool
2. Turn of the caching in pkg. (This will save you a lot disc space!) pkg set-property flush-content-cache-on-success True Downloaded packages are then no longer kept on your harddrive in /var/pkg/download
3. deactivate unneeded services / uninstall software. This is a personal one - Decide what tools you wanna have and which ones not (Presumably you can uninstall a lot of the languages packs). Some service which you could deactivate are: svc:/application/desktop-cache/input-method-cache:default svc:/application/desktop-cache/mime-types-cache:default svc:/application/desktop-cache/icon-cache:default svc:/application/desktop-cache/desktop-mime-cache:default svc:/application/print/ppd-cache-update:default Use the following command to do so svcadm disable ${svc} svccfg delete ${svc}
4. Find leaf/orphaned packages - This is more complicated but also a very important part. Especially some libs which you do not need any longer tend to stay. What you can do is the following - Find all the dependencies using /bin/pkg search -l \'depend::\' And then find all installed packages: pkg list All items which are installed but nobody depends on - are therefore leafs/orphaned. But be carefully - this will also list eclipse etc. But this is a way to find the orphaned libs. Personally I did some parsing with python to parse the two list of dependencies and installed packages to find the unique ones.

Overall these actions saved me 4Gb of space. Next thing to play with is tuning ZFS :-) Installing it on hybrid system. Have a look here: http://blogs.sun.com/brendan/entry/test

Last weeks HPC workshop has been great! Very interesting presentations and a lot of good talks. Nice to meet and great the people behind so many projects inside and outside of Sun working on Cloud, HPC, Sun Grid Engine, Service Domain Manager and similar! Slides can be found here: http://wikis.sun.com/display/SunHPC09. This includes my slides: http://wikis.sun.com/download/attachments/170755116/20090825_open_cloud_frameworks_hpc_tmetsch-v0.2.pdf

So happy to see the poster I made at the Sun HPC workshop:

Here is the talk I gave at GridKa School 2009 in Karlsruhe. Recordings will follow later – so stay tuned. The slides itself might not be to easy to read because they are designed for presentation not for Offline usage.

A transcript can be found here: http://85.114.139.198/nohuddleoffense/?p=369

Twitter gives you an unique chance to show your customers that your are actually improving your product. Twittering that bugs are fixed an when new features are added, tested and released helps keeping the noise down in forums etc. Also (future) customers can retweet or suggest additions. So to all management: Let your software developers twitter! It will help your product!

<link>

Take your current product and sell it as 'the Cloud'...:-)

Tweet: sometimes 140 chars isn't enough - new blogpost: http://bit.ly/HtbeB #cloudcomputing #OCCI

When I started listening to the audio book of 'Hector and the search for happiness' which is written by Francois Lelord I didn't expect much...But it was a very pleasant 5 hours car-ride when getting home. In total Hector the protagonist comes up with 23 lectures of what happiness is. I just wanted to add #24:

Happiness is when you know that you are happy

Just a short (meaning: very short) tutorial on howto create a OpenSolaris USB stick:

1. Install OpenSolaris in VirtualBox
   
2. Boot OpenSolaris and install the SUNWdistro-const package
3. run pfexec usbgen osol-0906.iso osol-usb.img /tmp
4. run pfexec usbcopy osol-usb.img