Availability Engineering
Sun Cluster Oasis
« Sun Cluster Geograph... | Main | Taming a Runaway... »
Wednesday Nov 01, 2006
Sun Cluster and Solaris Zones

I've been intrigued by Solaris Zones since I first heard about the project several years back. Thus, I was excited to be given the chance to help lead the team responsible for ensuring that Sun Cluster 3.2 take full advantage of this great new virtualization technology.

Now that Sun Cluster 3.2 is in beta, I'm pleased to report that we've done an outstanding job (if I do say so myself) of integrating Sun Cluster with Solaris Zones. Sun Cluster now supports zones as full-fledged “virtual hosts” of cluster services. Using zones with Sun Cluster is easy because the Sun Cluster resource management interfaces for using zones and physical nodes are identical. Practically, that means you can just specify a zone instead of a physical node in a Resource Group nodelist and the cluster will run the application inside that zone instead of in the global zone, with most of the isolation and security benefits thereof. And the best part is that the Sun Cluster agents (data services) don't need to be modified at all to run in the non-global zones. For example, the same HA-Apache agent that runs in global zones works perfectly in non-global zones.

So why should you care? Here are a couple reasons why you might want to use zones with Sun Cluster:

Now I'll provide a small example of how easy it is to run the HA-Apache service in non-global zones. This example assumes you've already configured apache properly in each non-global zone by providing an http.conf file, etc. Once you've done that, in order to add high availability, simply run the following cluster commands:

  1. Register the resource type

    # clrt register SUNW.apache

  2. Create the resource group. Note that the nodelist specifies three non-global zones using : syntax. The physical nodes are named ptriped1, ptriped2, and ptriped3. The non-global zones are named lztriped on each node.

    # clrg create -p nodelist=ptriped1:lztriped,ptriped2:lztriped,ptriped3:lztriped apache-rg

  3. Add the LogicalHostname IP address (named triped-2 on my system).

    # clreslogicalhostname create -g apache-rg triped-2

  4. Add the apache resource:

    # clrs create -g apache-rg -t apache -p bin_dir=/usr/apache2/bin/ -p port_list=80/tcp apache-res

  5. Bring the service online.

    # clrg online -M apache-rg

  6. Verify the status

    # clrg status

    === Cluster Resource Groups ===

    Group Name Node Name Suspended Status
    ---------- --------- --------- ------
    apache-rg ptriped1:lztriped No Online
              ptriped2:lztriped No Offline
              ptriped3:lztriped No Offline


    # clrs status

    === Cluster Resources ===

    Resource Name Node Name State Status Message
    ------------- --------- ----- --------------
    triped-2 ptriped1:lztriped Online Online - LogicalHostname online.
             ptriped2:lztriped Offline Offline
             ptriped3:lztriped Offline Offline

    apache-res ptriped1:lztriped Online Online - Service is online.
               ptriped2:lztriped Offline Offline
               ptriped3:lztriped Offline Offline

    And in the zone ptriped1:lztriped --

    # ps -ef | grep httpd
    webservd 3651 3646 0 17:17:36 ? 0:00 /usr/apache2/bin/httpd -k start
    webservd 3650 3646 0 17:17:36 ? 0:00 /usr/apache2/bin/httpd -k start
    webservd 3649 3646 0 17:17:36 ? 0:00 /usr/apache2/bin/httpd -k start
    webservd 3648 3646 0 17:17:36 ? 0:00 /usr/apache2/bin/httpd -k start
    webservd 3647 3646 0 17:17:36 ? 0:00 /usr/apache2/bin/httpd -k start
    root 3661 3407 0 17:18:06 pts/2 0:00 grep httpd
    root 3646 2393 0 17:17:35 ? 0:00 /usr/apache2/bin/httpd -k start

That's it! I hope you try out this great new feature when you get your hands on Sun Cluster 3.2!

Nick Solter
Sun Cluster engineering

Posted at 09:56AM Nov 01, 2006 in Sun  |  Comments[14]

Comments:

Any idea when SC3.2 will be released?

Posted by Ceri Davies on November 11, 2006 at 03:32 AM PST #

December '06

Posted by zoram on November 13, 2006 at 03:19 AM PST #

Great news, thank you.

Posted by Ceri Davies on November 13, 2006 at 03:26 AM PST #

I have two zones that are setup with
ip-type: exclusive.

I can't seem to add the Logicalhostname resource group.

Here is the error message I get.

# clreslogicalhostname create -g apache-rg triped-2
clreslogicalhostname: specified hostname(s) cannot be hosted by any adapter on global-host-01
clreslogicalhostname: Hostname(s): triped-2

Please help?

Posted by Wayne Dovey on January 14, 2008 at 09:02 AM PST #

Wayne,

Your clreslogicalhostname command is trying to create a logical hostname resource named 'triped-2'. triped-2 will also be the In order for this to succeed, the hostname 'triped-2' has to be defined on your local subnet(s) to which your cluster nodes are attached. That is, the hostname 'triped-2' along with an associated IP address needs to be defined in a hostname database such as /etc/inet/hosts or /etc/inet/ipnodes on each cluster node, or in the 'hosts' NIS maps accessible to cluster nodes, etc.

Instead of 'triped-2', use a hostname/IP address that is actually available on your local network and which is not already in use by another host.

Posted by Martin Rattner on February 18, 2008 at 09:15 PM PST #

One further note: the use of a LogicalHostname resource in non-global zones with ip-type=exclusive is not (yet) supported. Sorry I didn't notice this point previously. This feature is in development for a future release.

Posted by Martin Rattner on February 19, 2008 at 02:58 PM PST #

Hi,

Please can you point me in the direction of any instructions that explain how to set up a full virtual cluster on a single node. I do not have the equipment to build a physical cluster but would be interested to find out how I can set up a virtual cluster using zones on solaris 10.

We do this already on HPUX Itanium and it works very well. So as not to be outdone by my work mates I would like to show them I can do the same thing on Solaris.

Regards

Steve

Posted by Steve Luther on May 13, 2008 at 01:48 AM PDT #

@Steve
Neil's post titled New Solaris 10 Experience (http://blogs.sun.com/SC/entry/new_solaris_10_experience) walks through a more complete example of a virtual cluster with zones. If you don't have a spare disk slice to play with, you can put the failover zpool on a USB stick.

Posted by Nick on May 23, 2008 at 12:26 PM PDT #

I'm using a USB Flash Drive for a failover zpool but getting errors on zpool create.

Any ide on how to resolve? I tried using a USB Harddisk but I'm getting the same error.

free(fe6f6420): invalid or corrupted buffer
stack trace:
libumem.so.1'?? (0xff23b460)
libCrun.so.1'__1c2k6Fpv_v_+0x4
libCstd_isa.so.1'__1cDstdMbasic_string4Ccn0ALchar_traits4Cc__n0AJallocator4Cc___2G6Mrk1_r1_+0xb8
libCstd.so.1'__1cH__rwstdNlocale_vector4nDstdMbasic_string4Ccn0BLchar_traits4Cc__n0BJallocator4Cc_____Gresize6MIn0E__p3_+0xc4
libCstd.so.1'__1cH__rwstdKlocale_imp2t5B6MII_v_+0xc4
libCstd.so.1'__1cDstdGlocaleEinit6F_v_+0x44
libCstd.so.1'__1cDstdNbasic_istream4Cwn0ALchar_traits4Cw___2t6Mn0AIios_baseJEmptyCtor__v_+0x84
libCstd.so.1'?? (0xfe4fb164)
libCstd.so.1'?? (0xfe4fb8ac)
libCstd.so.1'_init+0x1e0
ld.so.1'?? (0xff3bfea8)
ld.so.1'?? (0xff3cca04)
ld.so.1'_elf_rtbndr+0x10
libCrun.so.1'?? (0xfe5ea950)
libCrun.so.1'__1cH__CimplKcplus_init6F_v_+0x48
libCstd_isa.so.1'_init+0xc8
ld.so.1'?? (0xff3bfea8)
ld.so.1'?? (0xff3c5318)
ld.so.1'?? (0xff3c5474)
ld.so.1'dlopen+0x64
libmeta.so.1'sdssc_bind_library+0x88
libdiskmgt.so.1'?? (0xff2a092c)
libdiskmgt.so.1'?? (0xff29a6b4)
libdiskmgt.so.1'?? (0xff29a42c)
libdiskmgt.so.1'dm_get_stats+0x12c
libdiskmgt.so.1'dm_get_slice_stats+0x44
libdiskmgt.so.1'dm_inuse+0x74
zpool'check_slice+0x20
zpool'check_disk+0x144
zpool'check_device+0x4c
zpool'check_in_use+0x108
zpool'check_in_use+0x174
zpool'make_root_vdev+0x3c
zpool'?? (0x1321c)
zpool'main+0x130
zpool'_start+0x108
Abort - core dumped

Posted by RMP on June 27, 2008 at 06:43 AM PDT #

Everything's fine. I just need to use

#NOINUSE_CHECK=1 zpool create -m /ZFSsamba HAZpool c4t0d0

Great and very helpful documentation you have here!

Posted by RMP on July 03, 2008 at 09:18 AM PDT #

From the Sun Cluster 3.2 documentation, there seems to be two options regarding Solaris Zones in a cluster. 1.) You can create zones, install the data service in the zone, and have the data service fail from one zone to the next. 2.) Solaris Zones HA, which makes a zone highly available. Of the two options, what would be the reason to deploy Zones in HA compared to deploying the application in the Zone and failing the data service from one zone to the next. What are the advantages of each method? Thanks in advance,

Ryan

Posted by Ryan DeWaele on January 29, 2009 at 06:41 AM PST #

Ryan,

To address your question, we have a section within the Concepts Guide that gives an overview of zones-as-nodes versus the HA Zones agent, and includes some recommendations:
http://docs.sun.com/app/docs/doc/820-4676/gcbkf?a=view

That information is based on an earlier presentation which is available on the HA Clusters community page:
http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/Technical-Presentations/

The direct link would be
http://opensolaris.org/os/community/ha-clusters/ohac/Documentation/Technical-Presentations/SunClusterAndSolaris10Container.pdf

In the forthcoming 1/09 update of Solaris Cluster, resource group affinities now default to being interpreted on a physical-node basis. This means that you can define affinities between a resource group that runs in non-global zones and a resource groups that runs in the global zone. One result of this is that you can effectively use the same zone in both of the modes that you have designated (1) and (2). The zone can fail over under the control of the HA Containers agent, and resource groups can then be started within the zone. The use of a '+++' RG affinity between the resource groups assures that they will fail over together.

The above-referenced material does not consider zone clusters, which are a new feature coming out in the Solaris Cluster 3.2 1/09 update. So there is actually a third way to use zones in Solaris Cluster. There will be a number of blogs forthcoming on the new features in this release, including zone clusters.

Posted by Martin Rattner on January 30, 2009 at 11:33 AM PST #

Thank you Martin for the explicit details. When more questions arise, I know were to ask. Thanks again!

Ryan

Posted by Ryan DeWaele on January 30, 2009 at 12:17 PM PST #

Hi Nick,

I tried HA-apache on my single solaris 10 node. But somehow the apache cannot be started when the cluster initially starts. Eventually it timed out after 5 min (initial start up will try more than 5 min). The problem is about the same reported in this link: http://forums.sun.com/thread.jspa?threadID=5350243

So my questions are:
1. do I have to install 3 copies of apache separately on global zone and my other 2 zones?
2. if not why apache could be started as I could see that apache_svc_start was launched but not httpd processes were started
3. how would I debug this issue? I mean how I could find useful logs other than /var/adm/messges.

I'd much appreciate your help.

Mike

Posted by mike xue on March 09, 2009 at 01:45 PM PDT #

Post a Comment:
  • HTML Syntax: NOT allowed
« Sun Cluster Geograph... | Main | Taming a Runaway... »