Hasham Pathan's blog

Deploying Apache HTTP Web Server on Sun Storage 7000 Unified Storage Systems

A a Sun BluePrints™ article authored by me explains details on Deploying Apache HTTP Web Server on Sun Storage 7000 Unified Storage Systems, and invoke Web services from the storage appliance. In addition, scenarios designed to aid consolidation efforts and simplify management are included. Readers should look into individual organizational requirements before considering the described solutions or other deployment options to see if additional product features can be of value. For example, several Sun Storage 7000 Unified Storage System capabilities, such as snapshots, cloning, and replication, often can further enhance infrastructure deployments.

Posted at 10:24AM Jun 10, 2009 by Hashamkha Pathan in Sun  | 

MySQL Cluster Database 7: Performance Benchmark

Summary of Results:

We recently conducted a Performance Benchmark of the newly  released version of MySQL Cluster database software version  7.0 using the DBT2 Benchmark. The results are very impressive. 

Highlight of the results:

• For 4 Node Cluster, we achieved 251,000 Transactions per minute which is more than 4X improvement over the MySQL Cluster 6.3 release.
• For 2 Node Cluster, we achieved 143,000 Transactions per minute which is more than  4X improvement over the MySQL Cluster 6.3 release.
• Whether a user is looking to take advantage of the latest multi-core, multi-thread server processors, or is seeking to reduce space and energy consumption of their Cluster hardware, MySQL Cluster 7.0 provides a higher throughput, more efficient solution than previous generations of the MySQL Cluster database.

Note that these results were obtained using 1GB  Gigabit Ethernet. We expect improved performance for high speed cluster interconnect like InfiniBand and Dolphinics interconnect solutions.  Testing using these interconnects is currently underway.

Some of the KEY features of MySQL cluster 7 include "ability to add nodes and node groups online"  and "Data node multithreading support" You can look at the list of new feature available in MySQL cluster 7 here. 

Deployment Architecture and Configuration Details:

See below for the deployment architecture details. In the case of a 2 node scenario, the data node processes were running on TWO Sun Fire x4450 system with 8 processor cores per data node. The MySQL server nodes were running a combination of Sun Fire x4450 systems and Sun Fire x4600 systems as shown in the deployment diagram below. 

In the case of a 4 node scenario, FOUR x4450 system were used to deploy the Data Nodes, each data node using 8 cores.  The MySQL Server nodes were running on a combination of TWO Sun Fire x4600, ONE Sun Fire x4240 and FOUR Sun Fire x4450 systems.(co-located with data node systems). Other relevant details are captured in the deployment topology diagram below.

The topology diagram for 2 Node Scenario :

The topology diagram for 4 Node Scenario:

Detailed Results:

FOUR Data Node:

TWO Data Node:

Important Notes:

1. Each MySQL server was bound to 3 processor cores using  "processor set" feature(to accommodate more MySQL Servers on the available hardware).  It is observed that we get increased  TPM/MySQL Server and slight decrease in the response time with 4 processor cores per MySQL server).  An example:  With 3 processor cores assigned for  the MySQL server process, we get the TPM of 12988 for 1 Server, 20 Warehouses.  With 4 processor cores, the TPM increases to 19113 with decrease in the response time from 52ms to 34ms.

2. TPM - New-order transactions per minute. It should be noted that the each transaction comprises of about 30 queries, so average response time per query if calculated would be  ~3ms. More details on the "transaction" are available in the TPC-C specification  document and DBT2 code base.

3. The MySQL 6.3 Cluster saturate with fewer number of MySQL servers. For comparison purpose,  we ran the tests with equal number of MySQL servers for both  MySQL Cluster 6.3 and 7.

Benchmark Description

Database Test 2 (DBT-2) :  DBT2 is an open source benchmark developed by OSDL (Open Source Development Labs ). Significant updates to the benchmark were made to simplify its ability to run with a clustered database such as MySQL Cluster Carrier Grade Edition. DBT2 simulates a typical OLTP (Online Transaction Processing) application that performs transactions with around ten to twenty SQL statements per transaction within five distinct transaction types. The DBT2 benchmark can be used as a good indicator in regards to the expected performance of any application that performs fairly simple transactions and executes these transactions in a repetitive fashion.

For the benchmark results above, DBT2 and MySQL Cluster were configured as an “in memory” database to simulate typical “real-time” database configurations. It should be noted that performance results are being measured as new-order transactions per minute (TPM).  The changes made to the DBT2 benchmark are documented and can be found on the SourceForge page for DBT2 downloads. It can also be downloaded from ww.iclaustron.com.

Additional Information:

More information on MySQL Cluster

MySQL Cluster 7 Architecture and New Features

MySQL Cluster Evaluation Guide

Benchmarking Highly Scalable MySQL Clusters - A While Paper published in 2007

Posted at 10:14PM Apr 21, 2009 by Hashamkha Pathan in Sun  |  Comments[3]

Unique Experience - EDUSAT & VTU

I delivered a session on Server Virtulization to  VTU(Visvesvaraya Technological University) students.  This was unique experience for me. For the first time,  I delivered a session that was LIVE broadcast over Satellite. The session was received by around 100 engineering colleges.  I should have the statistics on exact number of student attended in next few days. I understand the count was about ~1000. I will post the link to the presentation & stats once available.  I wasn't aware that the EDUSAT is actively used for e-learning by VTU. You can find more details here

Posted at 10:18PM Nov 11, 2008 by Hashamkha Pathan in Sun  | 

Using ZFS iSCSI LUN

ZFS - A New Approach to Data Management - Works on Pooled Storage Model and eliminates the need of volume manager. If you are new to ZFS, you may want to take look at ZFS resources here One of the important feature of ZFS that people don't talk as much is "iSCSI support". Steps below should help you quickly bring up the iSCSI configuration using ZFS. The setup here includes OpenSolaris 2008.05 running on dual core AMD system. This system will be configured to provide ZFS iSCSI volume(iSCSI target). A system running Solaris 10 is used as iSCSI client(initiator). In this case the "iSCSI client" is a virtual machine running Solaris 10 05/08 on the same physical system. Note that these steps should work seamlessly on any other system with Solaris 10.

High Level Steps:

1. Create a ZFS Volume (In existing ZFS pool OR Create a New Pool if you have spare disks )
2. Make Volume as it iSCSI target
3. Configure Solaris as iSCSI Client and access the iSCSI LUN
4. Create & Mount the File System on iSCSI LUN

Step 1: Create Volume: In existing ZFS pool OR Create a New Pool if you have spare disk(s)
Since I am running OpenSolaris 2008.05, it by default comes with ZFS as root file system which means we already have a default ZFS pool configured. We will create a new volume on this default pool. You can list the currently configured ZFS Pools & ZFS volumes using the commands below.
-bash-3.2# zpool list
NAME    SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
rpool   230G  19.8G   210G     8%  ONLINE  -
-bash-3.2#

-bash-3.2# zfs list -t volume
No volumes Listed

Create a new volume on default pool name "rpool".  Note the use of  "-V" switch and 2gb as size.
-bash-3.2#zfs create -V 2gb rpool/testvol
-bash-3.2# zfs list -t volume
NAME            USED  AVAIL  REFER  MOUNTPOINT
rpool/testvol     2G   207G  62.7M  -
-bash-3.2# 

Listing the properties of the newly created volume. Note the "shareiscsi" value. It's set to "off" by default

-bash-3.2# zfs get all rpool/testvol
NAME           PROPERTY         VALUE                  SOURCE
rpool/testvol  type             volume                 -
rpool/testvol  creation         Mon Jul 28 11:31 2008  -
rpool/testvol  used             2G                     -
rpool/testvol  available        207G                   -
rpool/testvol  referenced       16K                    -
rpool/testvol  compressratio    1.00x                  -
rpool/testvol  reservation      none                   default
rpool/testvol  volsize          2G                     -
rpool/testvol  volblocksize     8K                     -
rpool/testvol  checksum         on                     default
rpool/testvol  compression      off                    default
rpool/testvol  readonly         off                    default
rpool/testvol  shareiscsi       off                     default
rpool/testvol  copies           1                      default
rpool/testvol  refreservation   2G                     local

Step 2: Make Volume as it iSCSI target
Check if you already have any iSCSI target configured on the system using the below command.
-bash-3.2# iscsitadm list target
-bash-3.2#
No targets Reported. Now let's make the "new ZFS volume" as iSCSI target using the command below. Verify the settings for the volume.
-bash-3.2# zfs set shareiscsi=on rpool/testvol
-bash-3.2# zfs get shareiscsi rpool/testvol
NAME           PROPERTY    VALUE          SOURCE
rpool/testvol  shareiscsi  on             local

Verify that the ZFS iSCSI target is listed using the command below:
-bash-3.2# iscsitadm list target
Target: rpool/testvol
    iSCSI Name: iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa
    Connections: 0
Note Down the iSCSI name, we will need it at the time of the iSCSI Client configuration.

3. Configure Solaris10 system as iSCSI Client and Access the iSCSI LUN
Login As Root user. Verify the number of Disks Seen by the OS - You can use "format" command for this. Note the number of disk reported before the "iSCSI client" setup.
bash-3.00# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number):

As you can see, this system is reporting ONE Disk in "format" command output. Now let's proceed with the iSCSI client configuration. Note that you can configure iSCSI client to be in "discover mode" or "Static Mode". The steps below are for static mode. We are also assuming that this system wasn't configured to use iSCSI target. You may have to tweak these steps if you are already using iSCSI on your system.
bash-3.00# iscsiadm list static-config
No staitic configuration is reported. Add a static configuration Using the below command. Note that "iSCSI Name" provided and the IP address is the address of the iSCSI Target system.
bash-3.00# iscsiadm add static-config iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa,129.158.226.83

bash-3.00# iscsiadm list static-config
Static Configuration Target: iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726fa,129.158.226.83:3260
bash-3.00#

Enable the iSCSI client(initiator) in Static Mode using the below command:
bash-3.00# iscsiadm modify discovery --static enable

Note the detection of new device in system log messages.
bash-3.00# dmesg|tail -5
Jul 28 12:11:47 dhcp-cblr03-226-127 genunix: [ID 936769 kern.info] sd1 is /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28
Jul 28 12:11:47 dhcp-cblr03-226-127 genunix: [ID 265984 kern.warning] WARNING: Page83 data not standards compliant SUN      SOLARIS          1
Jul 28 12:11:48 dhcp-cblr03-226-127 genunix: [ID 408114 kern.info] /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28 (sd1) online
Jul 28 12:11:48 dhcp-cblr03-226-127 genunix: [ID 834635 kern.info] /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28 (sd1) multipath status: degraded, path /iscsi (iscsi0) to target address: 0000iqn.1986-03.com.sun:02:cad2198d-1048-4657-cf4f-aed99a4726faFFFF,0 is online Load balancing: round-robin
Jul 28 12:11:54 dhcp-cblr03-226-127 /sbin/dhcpagent[52]: [ID 732317 daemon.warning] accept_v4_acknak: ACK packet on e1000g0 missing mandatory lease option, ignored
bash-3.00#

Now let's run "format" command again to see if report the newly detected device. You may choose to use it as raw device or use it under volume manager control. The steps here partition the device and create a UFS file system.
bash-3.00# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
       0. c0d0 <DEFAULT cyl 2085 alt 2 hd 255 sec 63>
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c2t010000E0815BA8FC00002A00488D6A28d0 <DEFAULT cyl 1021 alt 2 hd 128 sec 32>
          /scsi_vhci/disk@g010000e0815ba8fc00002a00488d6a28
Specify disk (enter its number): 1
selecting c2t010000E0815BA8FC00002A00488D6A28d0
[disk formatted]...
truncated output...
format> p
Please run fdisk first.
format> fdisk
No fdisk table exists. The default partition for the disk is:
  a 100% "SOLARIS System" partition
Type "y" to accept the default partition,  otherwise type "n" to edit the
 partition table.
y
format> p
Truncted output.....
partition> p

Truncated output.......
partition> mod
Select partitioning base:
        0. Current partition table (original)
        1. All Free Hog
Choose base (enter number) [0]? 1
Truncated output.......

Do you wish to continue creating a new partition
table based on above table[yes]?
Free Hog partition[6]?
Enter size of partition '0' [0b, 0c, 0.00mb, 0.00gb]: 1gb
Enter size of partition '1' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '3' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '4' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '5' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '7' [0b, 0c, 0.00mb, 0.00gb]:

Okay to make this the current partition table[yes]? yes
Enter table name (remember quotes): "new"
Ready to label disk, continue? yes

Verify the parition table information:
partition> p
Current partition table (new):
Total disk cylinders available: 1020 + 2 (reserved cylinders)
Part      Tag    Flag     Cylinders        Size            Blocks
  0 unassigned    wm       1 -  512     1024.00MB    (512/0/0)  2097152
  1 unassigned    wm       0               0         (0/0/0)          0
Truncated output....
.....
partition> quit
format> quit
bash-3.00#

4. Create & Mount the File System on iSCSI LUN

Next step is to create a UFS file system on the iSCSI LUN. Note the device name passed to format command is iSCSI LUN slice 0 (s0) - The slice we created using the "partition menu" of the format command.
bash-3.00# newfs /dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0
newfs: construct a new file system /dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0: (y/n)? y
/dev/rdsk/c2t010000E0815BA8FC00002A00488D6A28d0s0:      2097152 sectors in 512 cylinders of 128 tracks, 32 sectors
        1024.0MB in 26 cyl groups (20 c/g, 40.00MB/g, 19456 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 81984, 163936, 245888, 327840, 409792, 491744, 573696, 655648, 737600,
 1311264, 1393216, 1475168, 1557120, 1639072, 1721024, 1802976, 1884928,
 1966880, 2048832
bash-3.00#

Mount the file system using the command below:
bash-3.00# mount /dev/dsk/c2t010000E0815BA8FC00002A00488D6A28d0s0 /mnt
bash-3.00# df -k /mnt
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c2t010000E0815BA8FC00002A00488D6A28d0s0
                      984911    1041  924776     1%    /mnt

We are now READY to use file system created on ZFS iSCSI volume. You may want to update "/etc/vfstab" file for file systems to be mounted automatically across system reboots.

Posted at 12:43PM Aug 04, 2008 by Hashamkha Pathan in Sun  |  Comments[1]

Sun xVM VirtualBox and Host Interface Networking – OpenSolaris 2008.05 as Host OS

The Sun xVM VirtualBox allows you to tun run unmodified OS as virtual machine. The current version available is 1.6.2. You can get it from here. It redirects you to Sun Download Center.

The Sun xVM VirtualBox is Type-II hypervisor which means you install it on top of the base OS – often called Host OS. You can then create virtual machines – often called Guest OS. If you are not familiar with the hypervisor concepts and terms used in the Server virtulization topic, you may want to look at this Sun Blueprint Article by Chien-Hua Yen

One of the cool thing about Sun xVM VirtualBox is that, it support all major operating systems as Guest and Host OS.

Networking & Guest OS:

NAT(Network Address translation) is the quickest and easiest way to get access to external network from the virtual machines(Guest OS). This access method is transparent to the users as it does not require any changes on Host OS or the Guest OS.

If you require access to the Guest OS from external network (e.g. remote access using SSH, NFS, Windows Share etc) NAT method is not sufficient. In such cases, you may want to look at “Host Interface Networking”. More information on this is available in the user manual for virtual box available here.

In nut shell, this process involves creating a virtual network interface and use of  "network bridge" concept. The steps below are for Sun xVM VirtualBox running on OpenSolaris 2008.05 as Host OS and Solaris 10 07/08 as Guest OS. Installing “Sun xVM VirtualBox” , and “Creating and installing Solaris Virtual Machine” steps are not included. I must admit, steps for installing Virtualbox on OpenSolaris and creating virtual machine with Solaris Guest were quite straight forward.

My observations is that, on some of the OpenSolaris Builds, the “setup scripts” don't work out of the box. You can use the below given steps for MANUALLY configuring the host based networking for your Guest OS. In this case, the Guest OS is Solaris 05/08 .The same should work for other Guest OS.(Host Configuration Steps remain the same)

Manually Configuring the Host based Networking:

Step 1: You need a MAC address for the virtual NIC to be created. One way to generate this is to Click on Virtual Machine Settings, Network Tab, Generate MAC. Note down the MAC address.

Step 2: Create the Virtual NIC on the HOST OS

-bash-3.2# uname -a

SunOS hasham-pc 5.11 snv_86 i86pc i386 i86pc

-bash-3.2# /usr/lib/vna nge0 00:e0:81:5b:a8:a9        <= Provide MAC Address from earlier Step

Verify the Virtual NIC is created.

-bash-3.2# dladm show-link

LINK CLASS MTU STATE OVER

nge0 phys 1500 up --

vnic0 vnic 1500 unknown nge0

Step 3: In order for this configuration to persist across reboot, create a startup script. You can use the simple script as given below. If you plan create multiple VNIC OR already have VNIC, this script can be enhanced to handle specific VNIC. Currently it deals with “vnic0” by default

-bash-3.2# cat /etc/rc3.d/S99vnic.sh

#!/usr/bin/bash

/usr/lib/vna nge0 00:e0:81:5b:a8:a9

Step 4: Go to Virtual Machine Settings(In this case Solaris 05/08), Network Tab and provide the details of the Virtual NIC. As shown in the picture below. Note that attached to default to NAT, it has been changed to “Host Interface”

Step -5: Boot your Guest OS – In this case, Solaris 05/08, and configure the Network Interface as DHCP or Static IP address. In this case, the Virtual NIC is as seen by Guest OS is e1000g0. One of the easy way to re-configure your network and system configuration FOR Solaris is to use “sys-unconfig” command. Note that this revert system system's configuration to an "as-manufactured" state, ready to be re-configured again including host name, networking, TZ, root password etc. I prefer configuring it bye editing/creating the configuration files.

You should be able use the VNIC in the same way for ANY other Guest OS. I have tested this with Windows XP as Guest OS and works fine. No changes in the HOST OS configuration steps. On Windows XP Guest OS, network configuration was changes to use DHCP. Based on your Network Interface card, you may be required to install driver for it in the Guest OS. 

See screen shots below for making changes to the Virtual Machine Configuration to use Host based Networking.

Windows XP Virtual Machine Screen shot - Intel NIC Listed under "My Network". Also access to/from other systems on the network works.

Posted at 09:16AM Jul 28, 2008 by Hashamkha Pathan in Sun  |  Comments[4]

Using DTrace in Zones - An Example

Posted at 01:04PM Jul 14, 2008 by Hashamkha Pathan in Sun  | 

Setting up MySQL Cluster using Solaris Zones

Introduction to the terms used:
Zone or Non global Zone: A zone is a virtualized operating system environment created within a single instance of the Solaris Operating System.
Global Zone: The global zone is both the default zone for the system and the zone used for system-wide administrative control
MGM Node: Manage(configure/start/stop etc) the other nodes within the MySQL Cluster
SQL Node: Allow access to cluster data
Data Nodes: Store Cluster data.

Major steps involved in the MySQL cluster setup in Solaris Zones:

1. Create Solaris Zones
2. Installing MySQL Cluster Software
3. Configuring & Testing MySQL Cluster

1. Create Solaris Zones
Example of creating a zone using command line given below.  When dealing with zones, we mainly use two Solaris commands. These are “zonecfg” for configure/modify/delete zones configuration & “zoneadm” command for install/un-install/boot/halt the zone If you would like speed up things and NOT type these commands for creating all four zones, you may skip this section and use the script provided to create multiple zones  section.

1.2 Create Zone Using command line
1.2 Creating multiple zones using the script

1.1 Create Zone Using command line:

Step 1.1.1: Configure the Zone
Step 1.1.2: Install the Configured Zone
Step 1.1.3: Complete the post-install tasks
Step 1.1.4: Accessing the Zone

1.1.1 Configuring the Zone:
# zonecfg -z mysql
mysql: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:mysql> create
zonecfg:mysql> set zonepath=/zones/mysql <== Replace this with the directory where you have space (Around 200MB for zone + Additional space for Application that you plan to install)
zonecfg:mysql> set autoboot=true <== Zone will autoboot with system reboot
zonecfg:mysql> add net
zonecfg:mysql:net> set physical=bge0<= Replace this with the interface name your system has(User #ifconfig -a to verify the interface name)
zonecfg:mysql:net> set address=10.12.161.101/24 <= Replace this with the IP address on the same subnet as Global Zone /24 is subnet mask)
zonecfg:mysql:net> end
zonecfg:mysql> verify
zonecfg:mysql> commit
zonecfg:mysql> ^D

Verify the Zone configuration status
#zoneadm list -icv  You should see newly configured zone entry + Global Zone

Step 1.1.2: Install the Configured Zone
# zoneadm -z mysql install
This step could take between 15-30 mins depending upon the speed of your Hard Disks/Processor speed

Step 1.1.3: Complete the post-install tasks 
From One Terminal Window - Connect to Console of the newly created zone using the command below
#zlogin -C mysql (Uppercase C)

Open Second Terminal Window and initiate a boot command as given below
# zoneadm -z mysql boot  Switch to Console Terminal Window and Answer the first time system configuration questions. Once the zone configuration is complete, it will go through reboot. Observe this from Console Window

Step 1.1.4: Accessing the Zone

A new command “zlogin” is available to access the zone from GLOBAL zone without supplying user name/password.
#zlogin zonename (e.g. zlogin mysql ).
This command allows login to zone name "mysql" without root password - You need to be "root" user in GLOBAL Zone to run this command. Other Access Methods like telnet, ssh, rlogin will work as well. Type zonename command to verify if we are global zone or non-global zone
#zonename
global                         <==== You are currently logged into global zone.(Base OS)
#zonename
mysql                         <==== You are currently logged into zone name “mysql”

1.2 Creating multiple zones using the script
Solaris Containers and Resource Manager Guide has script to create the multiple zones. It can be found here Save the script into a file called “zone-create.sh” & Run the script to create the 4 zones needed for MySQL cluster
usage: ./zone-create.sh <#-of-zones> <zonename-prefix> <basedir>
-bash-3.00# ./zone-create.sh 4 mysql /export/home            (Make sure there is no training / as the end of the basedir argument)

configuring mysql1
Truncated output
.........................
........................
booting mysql4

The above step will create & boot the four zones. Verify that all zones are configured and running state by using the below command:
-bash-3.00# zoneadm list -iv
ID NAME STATUS PATH BRAND IP
0 global running / native shared
2 mysql2 running /export/home/mysql2 native shared
4 mysql3 running /export/home/mysql3 native shared
5 mysql4 running /export/home/mysql4 native shared
6 mysql1 running /export/home/mysql1 native shared

Modify the zone configuration for all the zones to use the network interface and IP address. By default, the zone creation script doesn't assign any network interface or IP address to zones.
Follow the steps below for ALL the zones:

# zonecfg -z mysql1
zonecfg:mysql1> add net
zonecfg:mysql1:net> set physical=bge0
zonecfg:mysql1:net> set address=10.12.161.101/24
zonecfg:mysql1:net> end
zonecfg:mysql1> commit
zonecfg:mysql1> exit

# zoneadm -z mysql1 reboot

Next step is to reset the root password for the zones that we created using the script. The script sets “root” user password which is passed in encrypted form at the time of zone creation.
Remember the “zlogin” command that we described earlier in the section? You need to login using “zlogin” and reset the root password on all the zones. See steps below. You may also want to reset the default shell for root user to be “/usr/bin/bash”

Reseting Root  User Password for the zones:
# zlogin mysql1
[Connected to zone 'mysql1' pts/4]
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
# hostname
mysql1
# zonename
mysql1
# passwd root
New Password:
Re-enter new Password:
passwd: password successfully changed for root
# ^D
[Connection to zone 'mysql1' pts/4 closed]
# zonename
global

Change Root User Shell:
# usermod -s /usr/bin/bash root
UX: usermod: root is currently logged in, some changes may not take effect until next login.
# ^D
Reboot the zone using the following command:
#zoneadm -z mysql1 reboot

Repeat above steps for the remaining 3 zones. Note that you have to use unique IP address for each zone.

We now have four zones(see below) required for MySQL Cluster Ready to use

MySQL Cluster Component	Zone Name	IP Address
MGT Node	mysql1	10.12.161.101
SQL Node	mysql2	10.12.161.102
Data Node 1	mysql3	10.12.161.103
Data Node 2	mysql4	10.12.161.104

2. Installing MySQL Cluster Software

2.1 Download & Install the MySQL Cluster Software for Solaris x64
2.2 Setup "my.cnf" configuration for MySQL Server
2.3 Verify access to the MySQL Server
2.4 Edit ".profile" for root user

2.1 Download & Install the MySQL Cluster Software for Solaris x64. Download Link
Transfer the downloaded binary to all the zone. You can transfer the file using ftp to the individual zone and follow the steps below

Important Note: We have created "sparse root zone", /usr file system is shared as read only to non-global zones. The way mysql binaries are packaged and as per the install instructions, you need to create link "/usr/local/mysql" to  the directory where "MySQL binaries are located".  Since /usr file system is read-only from non-global zone for sparse root zone configuration, we use the below given workaround. You can achieve the same by playing with "mysql_install_db" script.

Ensure that we are in global zone
bash-3.00# zonename
global
bash-3.00# mkdir /mysql-cluster-gpl-6.2.15-solaris10-x86_64
bash-3.00# ln -s /mysql-cluster-gpl-6.2.15-solaris10-x86_64 /usr/local/mysql

<> Now we can proceed with the installation in the zone. Verify if you are logged in to required zone.
bash-3.00# zonename
mysql1
bash-3.00# pwd
/
bash-3.00# ls my*                              <===== MySQL Binaries transferred under "/" directory of zone
mysql-cluster-gpl-6.2.15-solaris10-x86_64.tar.gz
bash-3.00# gzip -d mysql-cluster-gpl-6.2.15-solaris10-x86_64.tar.gz
bash-3.00# tar -xf mysql-cluster-gpl-6.2.15-solaris10-x86_64.tar
bash-3.00# ls -ltd /usr/local/my*
lrwxrwxrwx 1 root root 42 Jul 1 19:09 /usr/local/mysql -> /mysql-cluster-gpl-6.2.15-solaris10-x86_64

bash-3.00# groupadd mysql
bash-3.00# useradd -g mysql mysql
bash-3.00# chown -R mysql:mysql /mysql-cluster-gpl-6.2.15-solaris10-x86_64
bash-3.00# cd /usr/local/mysql
bash-3.00# pwd
/usr/local/mysql

bash-3.00# ./scripts/mysql_install_db --user=mysql

Installing MySQL system tables...
OK
Filling help tables...
OK
Truncated output
...............
You can test the MySQL daemon with mysql-test-run.pl
cd ./mysql-test ; perl mysql-test-run.pl

2.2 Setup "my.cnf" configuration for MySQL Server
Copy Sample "my.cnf"  shipped with MySQL as configuration file for our environment
root@mysql1:[/] # cd /usr/local/mysql/support-files/
root@mysql1:[/usr/local/mysql/support-files] # cp my-small.cnf /etc/my.cnf

bash-3.00# ./bin/mysqld_safe --user=mysql &
[1] 26530
080701 19:24:28 mysqld_safe Logging to '/usr/local/mysql/data/mysql3.err'.
080701 19:24:28 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data

2.3 Verify  access to the MySQL Server
bash-3.00# ./bin/mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.1.23-ndb-6.2.15-cluster-gpl MySQL Cluster Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>mysql> exit
Bye

bash-3.00# pwd
/usr/local/mysql
bash-3.00# ./bin/mysqladmin shutdown

Edit ".profile" for root user
Include PATH to MySQL Binaries and Default Shell Prompt Change
PATH=$PATH:/usr/local/mysql/bin;export PATH      <=====PATH to MySQL Binaries added
PS1="\u@\h:[\w] # ";export PS1                                <=====default PROMPT changes.

Repeat the steps on other three zones before proceeding further. After end of these steps, we will have four zones  namely "mysql1", "mysql2", "mysql3" & "mysql4"


3. Configuring & Testing MySQL Cluster
3.1 Configuring the management node
3.2 Configuring the Data and SQL Nodes
3.3 Starting and Stopping the Cluster
3.4 Testing Cluster Operation

3.1 Configuring the management node
Follow the steps below to configure the management node:

root@mysql1:[/] # mkdir /var/lib/mysql-cluster
root@mysql1:[/] # cp /usr/local/mysql/support-files/config.small.ini /var/lib/mysql-cluster/config.ini
root@mysql1:[/] # cd /var/lib/mysql-cluster/
root@mysql1:[/var/lib/mysql-cluster] # vi config.ini                           <=========== Edit the file and make the changes as shown below [NDBD DEFAULT]
NoOfReplicas: 2
DataDir: /usr/local/mysql/data
FileSystemPath: /usr/local/mysql/data
# Data Memory, Index Memory, and String Memory
DataMemory: 600M
IndexMemory: 100M
BackupMemory: 64M
[MGM DEFAULT]
PortNumber: 1186
DataDir: /usr/local/mysql/data
[NDB_MGMD]
Id: 1
HostName: 10.12.161.101                                                         # <========== Replaced  localhost entry with IP address of MGT Node Zone IP
ArbitrationRank: 1
[mysqld]
Id: 2
HostName: 10.12.161.102                                                        # <========== Replaced  localhost entry with IP address of SQL Node Zone IP
[NDBD]
Id: 3
HostName: 10.12.161.103                                                       # <========== Replaced  localhost entry with IP address of Data Node-1 Zone IP
[NDBD]
Id: 4
HostName: 10.12.161.104                                                      # <========== Replaced  localhost entry with IP address of Data Node-2  Zone IP

Before we start MGT Node processes, let's finish the configuration on SQL Node & Data Nodes

3.2 Configuring the Data and SQL Nodes

Follow the below step for SQL Node & Data Nodes
Edit /etc/my.cnf and the following entries:
# Cluster Configuration/Information Regarding MGT Server
[mysqld]
ndbcluster
ndb-connectstring=10.12.161.101
[mysql_cluster]
ndb-connectstring=10.12.161.101

3.3 Starting & Stopping the Cluster
On the management node run the below commands:
If you are already not logged in to the management zone, login in using the below command from global zone
global# zlogin mysql1
[Connected to zone 'mysql1' pts/4]
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
root@mysql1:[/] #cd /var/lib/mysql-cluster
root@mysql1:[/var/lib/mysql-cluster] # ndb_mgmd -f /var/lib/mysql-cluster/config.ini
root@mysql1:[/] #

On the SQL Node run the below commands:
If you are already not logged in to the SQL node zone, login in using the below command from global zone
global# zlogin mysql2
[Connected to zone 'mysql1' pts/4]
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
root@mysql2:[/] # mysqld_safe --user=mysql &

On the Data Node run the below commands:
If you are already not logged in to the Data node-1 zone, login in using the below command from global zone
global# zlogin mysql3
[Connected to zone 'mysql1' pts/4]
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
root@mysql3:[/] # ndbd
root@mysql3:[/] #

global# zlogin mysql4
[Connected to zone 'mysql1' pts/4]
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
root@mysql4:[/]ndbd
root@mysql4:[/]
Verify the cluster status by running the following commands from Management node:

root@mysql1:[/] # ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Cluster Configuration
--------------------- <>[ndbd(NDB)] 2 node(s)
id=3 @10.12.161.103 (mysql-5.1.23 ndb-6.2.15, Nodegroup: 0, Master)
id=4 @10.12.161.104 (mysql-5.1.23 ndb-6.2.15, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.12.161.101 (mysql-5.1.23 ndb-6.2.15)

[mysqld(API)] 6 node(s)
id=2 @10.12.161.102 (mysql-5.1.23 ndb-6.2.15)
id=5 (not connected, accepting connect from 10.12.161.101)
id=6 (not connected, accepting connect from 10.12.161.101)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

ndb_mgm>
SQL Node & both Data Node are reported in the configuration and connected. We can now move to testing the cluster configuration.
Stopping the cluster: To shutdown the entire cluster: Use "ndb_mgm -e shutdown" from management node.(Remember you would rarely shutdown the entire cluster in real deployment scenarios.)
root@mysql1:[/] # ndb_mgm -e shutdown

3.4 Testing Cluster Operation
Let's connect to the Cluster database. You need to access the cluster database via "SQL Node". Login to SQL Node zone and connect to the database and create a sample table.
Later force failure of one of the data node and see if affect the data base availability.
root@mysql2:[/] # mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.23-ndb-6.2.15-cluster-gpl MySQL Cluster Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>use test;
Database changed

mysql> CREATE TABLE SetupDetails
    -> (ZoneName varchar(15),
    -> Mcomponent varchar(30),
    -> IP int(15)) ENGINE=NDBCLUSTER;
Query OK, 0 rows affected (0.78 sec)
mysql> desc SetupDetails;
+------------+-------------+------+-----+---------+-------+
| Field      | Type        | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| ZoneName   | varchar(15) | YES  |     | NULL    |       |
| Mcomponent | varchar(30) | YES  |     | NULL    |       |
| IP         | int(15)     | YES  |     | NULL    |       |
+------------+-------------+------+-----+---------+-------+
3 rows in set (0.01 sec)
mysql>

Inducing the failure of one of the data node
On the MGT Node, verify the cluster status before the failure:
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=3    @10.12.161.103  (mysql-5.1.23 ndb-6.2.15, Nodegroup: 0, Master)
id=4    @10.12.161.104  (mysql-5.1.23 ndb-6.2.15, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.12.161.101  (mysql-5.1.23 ndb-6.2.15)

[mysqld(API)]   6 node(s)
id=2    @10.12.161.102  (mysql-5.1.23 ndb-6.2.15)
id=5 (not connected, accepting connect from 10.12.161.101)
id=6 (not connected, accepting connect from 10.12.161.101)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

On the SQL Node Initial a session:
root@mysql2:[/] # mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.1.23-ndb-6.2.15-cluster-gpl MySQL Cluster Server (GPL)
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> use test;
Database changed
mysql> desc SetupDetails;
+------------+-------------+------+-----+---------+-------+
| Field      | Type        | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| ZoneName   | varchar(15) | YES  |     | NULL    |       |
| Mcomponent | varchar(30) | YES  |     | NULL    |       |
| IP         | int(15)     | YES  |     | NULL    |       |
+------------+-------------+------+-----+---------+-------+
3 rows in set (0.01 sec)
mysql>

Reboot One of the Data Node:
global# zoneadm -z mysql4 reboot
Verify the status of the Cluster from MGT Node. Note that 12.12.161.104 is not connected status:
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=3    @10.12.161.103  (mysql-5.1.23 ndb-6.2.15, Nodegroup: 0, Master)
id=4 (not connected, accepting connect from 10.12.161.104)

[ndb_mgmd(MGM)] 1 node(s)
id=1    @10.12.161.101  (mysql-5.1.23 ndb-6.2.15)

[mysqld(API)]   6 node(s)
id=2    @10.12.161.102  (mysql-5.1.23 ndb-6.2.15)
id=5 (not connected, accepting connect from 10.12.161.101)
id=6 (not connected, accepting connect from 10.12.161.101)
id=7 (not connected, accepting connect from any host)
id=8 (not connected, accepting connect from any host)
id=9 (not connected, accepting connect from any host)

Did it affect our data availability? Check from SQL Node. You will notice it doesn't affect the availability of the data.
mysql> desc SetupDetails;
+------------+-------------+------+-----+---------+-------+
| Field      | Type        | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| ZoneName   | varchar(15) | YES  |     | NULL    |       |
| Mcomponent | varchar(30) | YES  |     | NULL    |       |
| IP         | int(15)     | YES  |     | NULL    |       |
+------------+-------------+------+-----+---------+-------+
3 rows in set (0.00 sec)
mysql>

What if we reboot the other remaining data node as well??

Now you are good to start  using the MySQL Cluster with zones. You can simulating failures on the Cluster nodes. In the previous section we have seen  a scenario of failed data node while we were accessing the database from SQL Node. Since we are running MySQL Cluster Solaris Zones, we now have ability to tests various failure scenario's as if these are different physical systems. 

Posted at 09:29AM Jul 04, 2008 by Hashamkha Pathan in Sun  |  Comments[2]