BC, DR, HA, SC and MdM

Podcast zum Thema "Hochverfügbarkeit und Sun Cluster"

Vor sechs Wochen saß ich mit Constantin zusammen, um einen Podcast zum Thema "Hochverfügbarkeit und (Sun) Cluster" aufzunehmen. Das Interview dauerte etwas über 2 Stunden und ist jetzt ungeschnitten auf systemhelden.de verfügbar, auch als mp3.

Zu Beginn versuchen wir, ganz allgemein den Begriff Hochverfügbarkeit zu ergründen. Dann diskutieren wir die Begriffe Verfügbarkeit und Zuverläsigkeit und das Problem der vielen Neunen, über die in diesem Zusammenhang immer wieder geredet wird, leider fast immer, ohne die zugrundeliegende Mathematik zu verstehen. Aber keine Angst, wir machen hier nicht Mathematik, sondern veranschaulichen das Thema mit vielen Anekdoten aus dem täglichen IT-Leben. In der zweiten Hälfte gehen wir dann auf technische Möglichkeiten ein, Hochverfügbarkeit zu implementieren und diskutieren natürlich auch Sun Cluster, das ja gerade mit einem neuen Release 3.2 11/09 verfügbar wurde.

Und wie immer, wenn man über Hochverfügbarkeit redet oder schreibt, darf nicht unerwähnt bleiben, dass Hochverfügbarkeit nur zu einem kleinen Teil durch die Wahl der richtigen Technologie erreicht werden kann. Mindestens genauso wichtig sind People, d.h. gut ausgebildete und motivierte Mitarbeiter, und Prozesse, die wohldefiniert sind und auch trainiert und gelebt werden. Da fällt mir gerade wieder eine Anekdote ein - aber lassen wir das lieber heute. Also: Products, People, Processes - in Sun-Sprache. 

Zu diesem Thema wurde ich übrigens im Nachhinein nach Quellen gefragt, welche die 40:40:20 Regel belegen. Leider habe ich auch die Originalquelle, die angeblich von Gartner stammen soll, noch nie gesehen. Aber Constantin hat ähnliche Zahlen bei Gartner gefunden, die diese Aussage bestätigen. Auch eine Studie von Boeing - Statistik über Flugzeug-Katastrophen von 1959 - 2001 - sagt eindeutig aus, dass menschliches Versagen, d.h. Pilotenfehler mit oder ohne fehlende oder schlechte Prozesse, die Hauptursache für Katastrophen ist. Ähnliches sagt auch die Statistik auf planecrashinfo.com.

Beim nächsten Mal werde ich das Thema MTBF etwas präziser fassen. Und ich werde mich bemühen, keine Formeln zu erklären. Ich glaube, das geht über Audio nicht.

 Viel Spaß beim Zuhören

Hartmut

Posted at 09:46AM Jan 12, 2010 by Hartmut Streppel in Availability  |  Comments[0]

Upgrade to Sun Cluster 3.2 11/09 successful!

Sun Cluster 3.2 11/09 - aka 3.2u3 - is out. It is a fascinating new release with lots of new features. 

• HA ASM for single instance Oracle databases
• HA LDOM to handle LDOMs as blackboxes, like the HA Container agent deals with "flying" containers
• HDS Universal Replicator Support - to implement controller based replication within a cluster
• IPSec and SCTP support for scalable load-balancing data services
• more agents support for Solaris Container Clusters
• and much more
  

I upgraded my test cluster last week with the rolling upgrade method. And although I also installed all the patches for S10u8 it worked like a charm.

One thing to watch out for is the quorum server. It has changed the communication format with the cluster nodes. Therefore it is mandatory to remove the quorum server from all clusters using it. Do not forget to configure an alternate quorum device for the short period you need for the update, if you want to have a safety net. Then you upgrade the quorum server itself which takes about a minute or so. Finally add the quorum server back to the cluster as a quorum "device". Follow the instructions in the installation guide.

Have fun using it.

Hartmut

Posted at 10:12AM Dec 22, 2009 by Hartmut Streppel in Cluster  |  Comments[0]

Cloud Camp Munich - a short review

A month after CloudCamp Munich took place, it is time to write a short summary.

1. I really liked the format of the unconference. Usually there are so many experts in the audience that it is a pity if you do not get their expertise as content into the agenda. With this format we had experts in the unpanel, with some vendors, some endusers, some open source proponents, and we had them in the working groups. This is a great advantage over standard conference formats.

2. If in Germany, one would think, that the audience is rather shy. No, they weren't. No problem to find the people for the unpanel. No problem to get good questions, and excellent participation in the working groups.

3. The only problem I saw - at least for some of us - was that topics presented got lost due to the highly packed content and agenda. So, I did not really follow the presentations of the working groups, as I was still preparing the results of my own. A note taker would help here.

4. I really liked the 5 minute presentations. In order to pitch your point, you can use slides, but only as  a nice background. I really liked the presentation by Phillipe Huber from Symetriq. He boiled it down to the point, and had very good pros and cons for clouds.

5. I had the pleasure to moderate the HA working group. We had a lively discussion whether there is or will be high availability in the cloud. I think the consensus at the end was, that

• looking at the prominent cloud offerings today, you get what you pay for; i.e. the default availability that is built into the cloud you are using. In order to get better resiliency, disaster recovery options, one would have to invest into using an alternate cloud as backup, implement monitoring and failover etc.
• but there is potential, i.e. a niche, for cloud offerings with higher availability, and DR built-in, etc. But that would be only available with a premium price. In order to attract customers with higher requirements this is probably the way to go.

6. Private Clouds. There was the interesting point made, that there are no private clouds. Why? Because you would have to invet upfront. Clouds are "pay-per-use" by definition. Ok!

If you see this as a hard requirement, then this is true. But, there will be all kinds of environments that fall somewhere in between the traditional data center and a real cloud. I see private clouds as a nice means, to benefit from the technical advantages of cloud computing, especially in large enterprises, where CAPEX is paid by someone else :-)

7. I usually try to shy away from discussing security. But I remember the quote that "Cloud security is less a technical than a process issue".

To summarize: "The Cloud will be the next level of the Internet!" Sorry, I forgot who made this statement.

Hartmut

PS: See you at CloudConf Stuttgart on 25Nov09.

Posted at 11:43AM Nov 19, 2009 by Hartmut Streppel in Cloud Computing  |  Comments[0]

CloudCamp München - 20.10 18-23Uhr, Sun Microsystems, Heimstetten

Nachdem das CloudCamp in Frankfurt erfolgreich durchgeführt wurde, steht jetzt das CloudCamp in München vor der Tür. Am 20.10. von 18:00 - 23:00 treffen sich die Münchner Cloud-Interessierten in der Geschäftsstelle von Sun Microsystems in Heimstetten. Über die Webseite des Münchner CloudCamps geht es auch direkt Registrierungsseite.

See you there! Hartmut

Posted at 02:33PM Sep 30, 2009 by Hartmut Streppel in Cloud Computing  |  Comments[0]

25. Marathon du Medoc

You might have wondered, what MdM in my headline stands for. And I did not explain it since I started writing this blog. It is short for Marathon du Medoc. It takes place around the small village of Puillac, which is a twin city with Pullach, a place near Munich, where I used to live. Pauillac is the home of 3 of the 5 premier crus red wines, which are supposed to be one of the best in the world.

This marathon is one of the most famous ones, not for its sportive character and world records, but for its non-sportive character. It was founded as a means to promote red wine from the Medoc region north of Bordeaux. One of the important rules is "Non deguises, s'abstenir", which can be translated with "If you do not come in disguise, please stay at home". Another rule is, if you plan to run a new PB, stay at home. So, nearly everybody runs in a costume and everybody has time to chat, to listen to bands playing, to dance, and - one of the most interesting parts of this "race" - to test red wines.

Yes, testing red wines during a marathon. I did not dare to do this, when I ran this marathon for the first time 5 years ago. And it took me 4 years to fully understand what this marathon is all about. They provide fantastic food during the course. On the last 3km they serve, bacon, oysters, entrecote and ice cream. There is a special food tent behind the finish line that provides much more - it is just fantastic.

 I enjoyed this 5th participation and had lots of fun again. We were lucky this time, that the temperatures did not rise too much. It had only 20°C in the morning and 28°C in the afternoon. That is considered low in Southern France. I remember the worst race, where we had 36°C in 2007.

I wrote a short "race report" (in German).

If you plan to attend the next one, which will take place on September 11th, 2010, make a note in late January 2010 to check their website. The race is usually sold out in less than 24h.

Hartmut - back to real work now

Posted at 09:23PM Sep 28, 2009 by Hartmut Streppel in Personal  |  Comments[0]

Open HA Cluster Summit in San Francisco

On Sunday, May 31st, 2009, the first Open HA Cluster Summit was held in San Francisco. It started with a very interesting keynote by Professor David Cheriton from Stanford. Second part was a panel discussion about "High Availability: Today and Tomorrow", led by Eve Kleinknecht, poduct manager for Solaris Cluster, with panelists:

• Dr. Ira Pramanick, Google
• Tasso Argyros,  Aster Data Systems
• Mark Callaghan, Google
• Sreeram Duvur, Sun Microsystems
• and myself
  

See the video of the panel discussion here. One of the conclusions of this panel was, that High Availability is needed more than ever. The fact that many consumers view services that they use as being available anyway, (i.e. everytime they need them), has made the effort needed to make this reality, vanish.

Therefor I quoted this one article that I copied years ago, discussing the safety of airlines. And this one safety expert said, that even a very unsafe airline brings you  to your destination 364 days a year safely. The difference between a safe and an unsafe airline is important on the 365th day!

The other conclusion was, that more advanced standards for HA are needed. HA architectures today cover more than one cluster; they cover services spread around a data center, probably across several of them, running on different platforms, using different HA mechanisms. But there was also kind of a consensus, that these standards are hard to define and even harder to agree upon, as the past has shown.

Some nice pictures of the event are in Nick's blog entry. He also provides a link to another video, that has Meenakshi Kaul-Basu and Dan Roberts discussing Open HA Clusters with Amor Kwok. 

Have Fun

Hartmut Streppel

Posted at 11:40PM Jun 15, 2009 by Hartmut Streppel in Availability  |  Comments[0]

SC3.2 1/06 under Virtual Box

It took me a while to get my 2-node cluster on my laptop running in Virtual Box. Reason was not that Sun Cluster was difficult to install, but rather to find a decent and portable network configuration. That was with VirtualBox 2.1. I will add some information at the end of this entry on my workarounds to get this working.

When I read about the new features of VirtualBox 2.2 in the networking space, I thought, this is exactly what I need for my 2-node cluster on my laptop:  Host-only networking! That should solve my problems and it did.

The main problem of a cluster environment is that the typical cluster node is a static entity. It is not supposed to be carried around on a laptop - and, what is the central point - to work with dynamically assigned IP addresses and names. Clustered servers need to know their counterparts by name and address in advance! This is what Host-only networking delivers.

Now, what does my configuration look like?

- a Toshiba Tecra M9 laptop, running OpenSuse 10.3 for historical reasons

- 3 virtual machines running Solaris 10u6; two of them, the cluster nodes, with 896MB of memory, which allows for a decent machine; the third node runs a stripped down version of Solaris 10u6 with only 384MB; its only purpose at the moment is to serve as the quorum server; maybe it has to server as iSCSI target later, when I decide to use shared storage, which I don't at the moment.

Now to the tricky part, the networking configuration. The 2 cluster nodes use 3 network interfaces, all configured as Intel Pro/1000. Adapter 1 is configured as "Host-only network", which is the new 2.2 feature; it is bound to adapter vboxnet0, which is automatically created on my guest. On my system it is assigned IP address 192.168.56.1 . Adapters 2 and 3 are configured as "Internal Network" with id's ic1 and ic2. You can make a guess what these are being used for. Solaris internally these 3 apapters are mapped to e1000g[0-2].

If you need access to the external network beyond your host, you must add a fourth adapater configured as "Bridged Network".

How do I configure the network on the Solaris hosts? e1000g0 is configured as DHCP. Virtual Box will assign an IP address to it and it will use the given nodename, which is exactly what I need. As the assignement of the DHCP-based addresses seems to be fixed, probably based on the (virtual) MAC addresses, I can add these addresses into /etc/hosts safely. I have an additional hostname.e1000g0 file that sets the network mask and puts the adapter into the appropriate IPMP group.

hartmut11 netmask 255.255.255.0 group sc_ipmp0 -failover

I assume that the hostname gets overwritten by DHCP.

I do not add any configuration data for the other 2 adapters as they will be used for the cluster interconnects. If you have configured an adapater to get access to the external network you have to configure it as DHCP. If the only thing you want from "externally" is using Sun Cluster Manager, then the "Host-only" network is sufficient.

Now I am all set to install and configure Sun Cluster 3.2 1/09. Without going into much detail, as this is a straight forward install, I attach the iso image as a DVD, cd to the directory with the JES installer in it and start it. Configuration will be done later.  The only interesting part during the install is to tell scinstall to use e1000g1 and e1000g2 for the cluster interconnect. That's it.

As I said, I am using a 3rd Solaris VM as my quorum server. But that is also just another simple Sun Cluster package install, without any major configuration work. I just changed the name of the QS instance.

Done! Now, I can do my testing. As I said, this cluster does not use shared storage. This needs some more work. But for my purposes, testing patching, this is sufficient.

Have fun

Hartmut Streppel

PS: I had promised to describe briefly what I had done prior to VirtualBox 2.2. OK, I had configured a 3rd internal network that I used solely for internode and quorum server communication. That was easy. I had configured my primary interface as bridged, which bound it to my eth0 network and gave me DHCP addresses and names, which was the problem. As long as I ran this environment in the same network, the DHCP assigned names remained the same. Attaching the laptop to my home DSL router changed the addresses and I had to edit /etc/hosts again and again.

What was worse is that DHCP also sets the nodename. As my cluster had been configured with fixed nodenames, and SC relies on them, I had to find a way to circumvent this problem. You can probably prevent this from happening but I used a trick, by adding a startup script that changed the nodename later in the boot sequence to keep the cluster happy. It worked but is not recommended.

Posted at 02:52PM Apr 23, 2009 by Hartmut Streppel in Cluster  |  Comments[1]

How to use one quorum server instance from multiple clusters

Sometimes my thoughts are too complicated. Maybe this is because I am an engineer - then everything has to be complicated. Or is it, because I work too much with engineers and know how they think?

 Anyway, I found out that I used a method, way too complicated, to add a quorum server to my test clusters. My idea was, that I needed a dedicated port for every cluster that wanted to use a quorum server. So for every additional cluster that needed a quorum server, I added a new line in the quorum server config file and started that new quorum server instance. It worked well and it is a valid configuration, but, as I said, too complicated. You can use just one quorum server instance, e.g. using the default port 9000 for any number of clusters.

Example: my configuration (/etc/scqsd/scqsd.conf) has the following 3 lines:

/usr/cluster/lib/sc/scqsd -i darling -p 9000 -d /var/scqsd
/usr/cluster/lib/sc/scqsd -i hvb -p 9001 -d /var/scqsd1
/usr/cluster/lib/sc/scqsd -i streppel -p 9002 -d /var/scqsd2 

As you can see, I am using 3 instances listening on ports 9000-9002, one for each of my test clusters. Now I changed my cluster "streppel", running under Virtualbox on my laptop to use the instance darling on port 9000. To prove that it works I did a "clqs show" on the quorum server darling. As you can see, there are now entries from 2 clusters under port 9000.

[darling:root] clqs show
=== Quorum Server on port 9000 ===
  ---  Cluster planets (id 0x498AFCE8) Reservation ---
  Node ID:                      2
    Reservation key:            0x498afce800000002
  ---  Cluster planets (id 0x498AFCE8) Registrations ---
  Node ID:                      1
    Registration key:           0x498afce800000001
  Node ID:                      2
    Registration key:           0x498afce800000002

  ---  Cluster streppel (id 0x49848C5C) Reservation ---
  Node ID:                      1
    Reservation key:            0x49848c5c00000001
  ---  Cluster streppel (id 0x49848C5C) Registrations ---
  Node ID:                      1
    Registration key:           0x49848c5c00000001
  Node ID:                      2
    Registration key:           0x49848c5c00000002

=== Quorum Server on port 9001 ===
  ---  Cluster hvb (id 0x485A6E5D) Reservation ---
  Node ID:                      1
    Reservation key:            0x485a6e5d00000001
  ---  Cluster hvb (id 0x485A6E5D) Registrations ---
  Node ID:                      1
    Registration key:           0x485a6e5d00000001
  Node ID:                      2
    Registration key:           0x485a6e5d00000002

=== Quorum Server on port 9002 ===
Quorum server on port "9002" is not configured in any cluster.
 

The only potential minor problem you can run into is, if you have several clusters with the same name. Not, that this causes any trouble with the cluster. The quorum server uses the unique id, assigned to each cluster to sort things out. But for the admin the output of "clqs show"
could be slightly confusing, as cluster id's are hardly used - at least I usually do not pay attention to them.

So, to summarize, you can use one quorum server instance to serve as quorum for multiple clusters. Good naming conventions help diagnosing potential problems.

Hartmut Streppel

Posted at 03:59PM Mar 26, 2009 by Hartmut Streppel in Availability  |  Comments[0]

Why a logical IP is marked as DEPRECATED?

The DEPRECATED flag and IP routing in a Sun Cluster environment!

IP routing is a complicated matter; it's not always as you think. One common misperception with logical IP addresses, i.e. IP addresses that are controlled by Sun Cluster THAT can migrate between servers is, that the response packet from an HA service must always have its logical IP address as source address. That this is usually not the case is the main part of my blog entry in the Sun Cluster engineering blog.

 Have fun reading it. Comments are welcome.

Hartmut Streppel

Posted at 09:35PM Mar 02, 2009 by Hartmut Streppel in Availability  |  Comments[0]

Sun Cluster 3.2 1/09 available

What a coincidence! I start my own blog and a new Solaris Cluster release shows up. Or is it the other way round?

Doesn't matter! The new release has a number of great features. Personally I like the flexibility features best, as they allow for even more and easier configurations in the field. And that's where I work. Those features include:

• Optional Fencing
• Software Quorum
• Optional dedicated partition for /globaldevices

I'll test all of these over the next couple of days on my new 2 node cluster, running in Virtualbox. And I'll let you know what I found out.

The most important new feature, though, is the zone cluster, the first virtual cluster, probably unmatched in the industry. It allows for running several, independent clusters under one "real" cluster, running in a Solaris global zone. I am wondering what use cases our users will find for this feature.

If you want to read more, you should go to www.sun.com/cluster.

Or you should have a look at the OASIS blog, that has some pointers to some short videos, where you can see and listen the some of the engineers explaining their technologies.

That's it for my first blog entry. Stay tuned and come back.

Posted at 10:21PM Jan 27, 2009 by Hartmut Streppel in Cluster  |  Comments[0]