A geek's geek-log My not-so-secret diary

Although it's the field I find most enticing, it's kind of strange I've never blogged about high availability clustering. Ever since my HOD, Professor M.S Gaur, asked me to deploy Linux HA (Heartbeat) to keep our servers highly available, I've developed a deep interest towards this concept. Disaster can strike at any time and when this leads to disruption of a service, an organisation can incur huge losses. To overcome this, we make use of high availability clustering where several nodes appear as a single entity to a client, offering the given service. The nodes together may either act as a failover cluster wherein only one node provides the service at a time (active node) and if it fails, another node rises to take it's place or it may be a scalable cluster where the load is distributed across multiple nodes to maintain availability. Here, I'll be giving a step by step detail on how to failover apache between zones.

The first step, of course, would be to install SXCE. I choose SXCE build 86. There isn't much to specify here apart from the fact that you need to create a 512MB partition to be mounted as /globaldevices. The cluster nodes will access devices connected to other nodes through here. Next, you have to set up a single node cluster. I won't be going into details of that because we have this excellent piece of documentation over here:

http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/SCXdocs/installsinglenode/

Now, we move on to the interesting part.

The plan here is to set up two zones named node1 and node2. We will be failing apache over the two of them.

#: zonecfg -z node1 create
zone3: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:node1> create
zonecfg:node1> set zonepath=/export/home/lalith/zones/1
zonecfg:node1> set autoboot=false
zonecfg:node1> add net
zonecfg:node1: net> set address= <IP 1>
zonecfg:node1: net> set physical= <device name>
zonecfg:node1> end
zonecfg:node1> verify
zonecfg:node1> exit

#: zoneadm -z node1 install

This will install zone node1. Now to configure node2.

#: zonecfg -z node2 create
zone3: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:node2> create
zonecfg:node2> set zonepath=/export/home/lalith/zones/2
zonecfg:node2> set autoboot=false
zonecfg:node2> add net
zonecfg:node2: net> set address= <IP 2>
zonecfg:node2: net> set physical= <device name>
zonecfg:node2> end
zonecfg:node2> verify
zonecfg:node2> exit

#: zoneadm -z node2 install

Now boot both the zones.

#: zoneadm -z node1 boot

(configure node1)

#: zoneadm -z node2 boot

(configure node2)

Apache by default won't have the httpd.conf file ready for you. So what we'll be doing here is to create a copy of the httpd.conf-example file with the name httpd.conf in the global zone and the non global zones.

In the global zone, zone node1 and zone node2,

#: cp /etc/apache/httpd.conf-example /etc/apache/httpd.conf

Now your cluster should have a logical hostname that will identify the currently active device. This will remain unique to the cluster. It is important that all devices in the cluster have this hostname listed in it's /etc/inet/hosts. I choose the name 'node' for my cluster (I just realised it doesn't make much sense :P )

In non-global zone, zone node1 and zone node2,

#: echo '<Cluster IP> node' >> /etc/inet/hosts

Now your apache is all ready to go. The next step would be to setup the resource groups for your single node cluster. First, you'll need to setup the logical hostname resource that will set the hostname (that you'd added into your /etc/inet/hosts file) to identify your cluster. It is essential that you do this otherwise your IP won't failover (not like the cluster will allow you to make the resource group without this :P). Second, we'll configure apache to run on our single node cluster. We'll be using the Java Web Console for this (if you haven't enabled it, look into the 'how to setup a single node cluster' documentation again).

First, login to the web console (http://127.0.0.1:6789 by default), and go to Sun Cluster.

The first step involved in making the cluster would be to setup a resource group. In the left menu, go to resource groups and then, select 'New'. You will get another window where you'll be asked details about your resource group. I've already made the resource group named web1, so my window would look different from yours. This is just to give you an idea of the details.

Now select the primary nodes, we'll be including our zones here.

Once you've setup your resource group, it's time to setup individual resources. Again, select resource groups and from the drop down, choose the resource group that you just made. Select 'New', and setup two resources, one for your logical hostname...

... and the other for your Apache webserver. Set logical hostname as a strong dependency and specify the binary path for your apache (/usr/apache/bin on mine).

I'll remind you again that I'm showing you the details AFTER I've set up my resources, so hence, the drop downs are missing and the menus will be different.

Our resource configuration is done. Now go back to the resources group menu, select the group we just configured and click on 'restart'. Check if the services are online. If Murphy is kind enough, your cluster should be good to go!

Next, we'll invoke a failover between our zones. First, let's get an idea of our cluster's status with the following command,

#: /usr/cluster/bin/cluster status

Here's the output on my system...

Notice that our first zone, mercury:node1 is now online. You can verify this by opening your web browser and entering in your address bar both the cluster IP and node1's IP. Both will show you apache's default test page which implies that node1 is now in charge of the cluster. Now, to invoke the failover, we shut down node1 with this command,

#: zoneadm -z node1 halt

#: /usr/cluster/bin/cluster status

And the output is.... 

There you go! You've just failed apache over to zone mercury:node2. You may verify the same using your web browser. Keep repeating the process as many times as you want. Don't forget, have fun!

 Cheers!