LaoTsao's Weblog (老曹的網路記)

ZFS

One of best feature in Solaris 10 is ZFS

ZFS was introduced in Solaris 10 update 2 , but ZFS is very young FS, there are many advancement in opensolaris and some are backported in Soalris 10.

Recently the intruducation of Openstorage and Unified Storage Appliance future introduce features that only presented in Unified Storage Appliance.

• ZFS Configuration Guide
• ZFS Best Practice Guide
• ZFS Evil Tuning Guide
• ZFS Troubleshooting Guide
• ZFS for Database

IMHO, Sun should speedup putting features/fixes back in Solaris 10, so many enterprise customer can take advantage of ZFS features.

Key features:

• user quota, right now one need separate FS for user quota and become issues when one need to setup quota for thousands users

Posted at 09:18AM May 17, 2009 by hstsao in Solaris 10  |  Comments[0]

coolstack, webstack and solaris zone

This is concern on AMP and solaris zone

coolstack 1.3.1 has version on sparc and x86, webstack also has version on sparc and x86

coolstack install pkg under /opt/coolstack

webstack install pkg under /opt/webstack

opensolaris 2008.05 only support x86, next release will support sparc.

the amp, amp-dev install apache(apache2) under /usr, MySQl under /usr/mysql and php under /usr/php etc

the blastware.org also provide amp and install pkg under /opt/CSW

As for AMP in solaris zone, the configuration file and some docs files are under /etc and /var so it is possible to use the spare zone and setup different zone with various different zone for various AMP.

Even under /opt , one can share /opt with various zone

Please see the following links:

coolstack 1.3.1
web stack 1.4

please see this link for MySql in solaris containers

some example of apache in zone

Posted at 07:50AM May 17, 2009 by hstsao in Solaris 10  |  Comments[0]

sort in 7bit and 8bit

IHAC wrote to me that depend on which subnet he login to T5120 the timex sort file>file.out on /tmp give very different result;
one is about 8 sec
one is about 2. minute 50 sec
a factor of 21 difference.

Turnout that the locale is different in two way of login to the system one set LANG=C
one setto LNAG=en_US.utf8
the result for LANG=en_US is similar to LANG=C

Posted at 04:40AM Jul 28, 2008 by hstsao in Solaris 10  |  Comments[0]

Flextronics IB switches

Flextronics provide some very inexpensive IB switches
infiniband for Mass Interesting article that talk about in expensive IB for HPC cluster

Posted at 06:21AM Jul 16, 2008 by hstsao in HPTC  |  Comments[0]

3 socket AMD system

Base on today HT-1.0, it seems that 3 socket AMD 8xxx CPU with the 3 HT-links will be a very balance IO server
with 3 HT links, each CPU will have HT link to other CPU and one HT link to IO chipset.

Posted at 09:22AM Jul 15, 2008 by hstsao in HPTC  |  Comments[0]

sb6048 IB-NEM switch connection without IB swicth

3x24 datacenter is perfect partner with sb6048 IB-NEM IB switch
What is the connection possibility for IB-NEM without a iB Switch?
Each IB-NEm contaions two 24 x4 port swicth chipset and provide dual connection to the 12 blades in one of the sb6048 chassis.
The other 2 12 port are presented as two set of 4 x12 port for outgoing connections.
The 12 blades in chassis are connected with fullly no blocking CLOS network.
betwenn the chassies the followings are possible:

• 4 x12 cables between two chassis for 24 blades for non blocking CLOS
• 2 x12 cables between 4 chassis for 48 blades for 50% blocking CLOS
• 1 x12 cables between 5 chassis for 60 blades for 25% blocking CLOS

Posted at 10:45PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

3x24 datacenter IB switch and sb6048

Sun recently introduce 3x24 Datacenter IB switch it is based on three 24 port x4 chipset
the key inovation is that it present the 72 x4 port as 24 x12 port in 1U form factor
It is best partner with the leaf switch IB-NEM in the Blade 6048
For smaller network (<=288) one could use 3x 24 datacenter swicth to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require single 3x24 datacenter switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 2 3 x24 datacenter switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 4 3 x24 datacenter switches

Posted at 10:30PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

x41450 and x4450 support of window 2008 server

The X4150 and X4450 are Certified to run Windows 2008 today(may/01/2008). There are no additional drivers needed for these platforms to work with Windows 2008. However, there is a minimum BIOS requirement
X4150 Mircosoft Windows 2008 Certification link: x4150 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:
X4450 Mircosoft Windows 2008 Certification link: x4450 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:

Posted at 03:11PM May 01, 2008 by hstsao in General  |  Comments[0]

rocks 5.0 released

download; http://www.rocksclusters.org/wordpress/?page_id=82 new features:

• Xen Support:
  You can use the Xen Roll to create VM Containers: physical machines that are used to hold Xen-based virtual machines. The Rocks Command Line was expanded to help build and maintain VMs (e.g., rocks create host vm compute-0-0-0? is used to install a VM).
• Fully-Programmable Partitioning
  The partitioning of client nodes (e.g., compute nodes and tile nodes) has been retooled. You can supply Red Hat partitioning directives to any node by writing a program in the pre section which populates the file /tmp/user_partitioning_info. The program can be as simple as small bash script that echos Red Hat partitioning directives or as complex as a python program that outputs partitioning info based on: the nodes name, the nodes membership, the number of disks in the node or the type of disks in the node. See the Base Roll documentation for details.
• OS: Based on CentOS release 5/update 1 and all updates as of April 29, 2008
• Condor: updated to v7.0.1
• Ganglia: Ganglia Monitor Core updated to v3.0.7
• Ganglia: phpsysinfo updated to v2.5.4
• Ganglia: rrdtool updated to v1.2.23
• HPC: The HPC roll is now optional. You can build a bare-bones cluster by using only the Kernel Roll, OS Rolls (disk 1 and 2) and the Base Roll
• HPC: MPICH updated to v1.2.7 patch 1)
• HPC: MPICH2 added to the roll (v1.0.6 patch 1)
• HPC: OpenMPI added to the roll (v1.2.6)
• SGE: SGE updated to 6.1 update 4
• SGE: Added tight integration for SGE and OpenMPI
• Area51: chkrootkit updated to v0.48
• Area51: tripwire updated to v2.4.1.2
• Bio: Biopython updated to v1.45
• Bio: Clustalw updated to v2.0.5
• Bio: Fasta updated to v35.3.5
• Bio: NCBI toolbox updated to Mar 2008 version
• Bio: MpiBlast updated to v1.5.0-pio and is patched against the NCBI toolbox Mar 2008 version
• Bio: Phylip updated to v3.67
• Bio: T_coffee updated to v5.65
• Bio: Gromacs and MrBayes are now MPI Enabled and compiled against rocks-openmpi

Posted at 07:25AM May 01, 2008 by hstsao in HPTC  |  Comments[0]

IB network design leaf and core switch trade-off

Current IB switch based on Mellanox IS III 24 port 4x chipset support upto 288 in 3 stages and 3456 in 5 stages.
Recently Sun has build in leaf switch in the Blade 6048 rack.
If one use 3stages IB switches with sb6048 build in leaf switch one will create 5 stages network, and if one use 5 stages IB switch then we will create 7 stage network.
TACC support more than 3456 nodes so it need 2 3456 switches and it use IB-NEM to create 7 stage CLOS network
For smaller network (<=288) one could use many 24 port switches as core swicthes to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require 3x24 port core switches
• 8 chassises of 12 blades: 2 sb6048 = 96 blade require 4x24 port core switches
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 6 x24 port core switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 12 x24 port core switches

5 stages network using NEM IB leaf switch

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

3 stages network using IB PCI-E EM :

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

For much larger network (>288) one could use many 288 port switches as core switches to build a 5 stages IB network. one can also use the dual port IB EM modules connect to core switches , for <=288 of 3 stages network and <=3465 for 5 stages networks.
Of course one use many core switches to construct larger IB network.
The same argument can apply to rack with rack server:
there are design tradeoff with big IB switch and has small leaf switches in each rack.
for examples if we use 36 1U server as building block and 3 24 port leaf switches in a rack , then there are two way to build IB network

• use 3 stages core switches (<=288) to build a 5 stages network
• use many 24 port switches as core switch for 3 stages network

The main difference between 3 stages and 5 stages IB network is the two switch hub difference: ~2x200 nanosecond difference for 24 port switch chipset.

Posted at 07:36PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

mellanox InfiniScale IV architecture

Mellanox announced InfiniScale IV architecture in sc2007
the benefits are:

• 40Gb/s server and storage interconnect (4x QDR)
• 120Gb/s switch-to-switch interconnect (12x QDR)
• 60 nanosecond switch hop latency
• 36-port switch devices for optimal scalability
• Adaptive Routing to optimize data traffic flow
• Congestion control to avoid hot spot

Using 36 port silicon to build an non-blocking CLOS switches, in 3 layers setup, one can get(for 24 port chipset we can get 288 port in 3 stages):

• 72 ports: 6 chips, 9 links between chips, 2 cores and 4 leafs
• 108 ports: 9 chips, 6 links between chips, 3 cores and 6 leafs
• 216 ports: 18 chips, 3 links between chips, 6 cores and 12 leafs
• 324 ports: 27 chips, 2 links between chips, 9 cores and 18 leafs
• 648 ports: 54 chips, 1 link between chips, 18 cores and 36 leafs

For 5 stage non-blocking CLOS network one can get (24 port chipset we can get to 3456 port in 5 stages)

• 2 x 648 = 1296 port
• 3 x 648 = 1944 port
• 4 x 648 = 2592 port
• 6 x 648 = 3888 port
• 9x 648 = 5832 port
• 12 x 648 = 7776 port
• 18x 648 = 11664 port

as comparison with InfiniScale IV we list the InfiniScale III 24 port features here:

• Twenty-four 10 or 20Gb/s Infiniband 4X ports or eight 30 or 60Gb/s Infiniband 12X ports (or any combination)
• 480Gb/s (SDR version) or 960Gb/s (DDR version) of total switching bandwidt
• Scalable to thousands of Ports
• 96 integrated 2.5Gb/s (SDR version) or 5Gb/s (DDR version) SerDes interfaces (physical layer)
• Auto-negotiation of port link speed
• Ultra low latency cut-through switching (less than 200 nanoseconds) MTU Size from 256 to 2K bytes
• Supports Multi-Protocol Applications for Clustering, Communication, and Storage
• Integrated Subnet Management Agent (SMA)

Posted at 05:38PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

Solaris Operating system Life Cycle

This import link describe the solaris OS life Cycle Sun's Life Cycle Model for the Solaris Operating System
Please note that market conditions and requirements can cause the exact date of the transition to vary
Key day to remember:

• Solaris 9 launched in May 2002
• Solaris 10 launched in January 2005

Posted at 05:54AM Apr 17, 2008 by hstsao in Solaris 10  |  Comments[0]

x2200M2 QC U.S. Education Essentials Matching Grant

This year MG has very aggressive 60% discount on most products The following x2200 M2 QC is particular interesting for HPC customer

• A85-FFZ2-H-8GB-JL8
• Sun Fire X2200 M2 x64 Server: 2x Quad Core AMD Opteron Model 2354 processors (2.2GHz/95W, B3), 4x 2GB registered ECC DDR2-667 memory, no hard drives, no optical drive, 1x PSU, Service Processor, 4x 10/100/1000 Ethernet ports, 6x USB 2.0 ports, 1x I/O riser card with 2x PCI-Express x8 slots, no power cords, order Geo-specific x-option. RoHS-5. Standard Configuration.
• list $2,905.00
• MG $1,162.00

Posted at 10:27PM Apr 16, 2008 by hstsao in HPTC  |  Comments[0]

change hostid

Recently a customer request some way to migrate from older Sun server and solaris to newer server and Solaris but with one condition: need to preserve the hostid so the license server can use the license file with old hostid.
After some research, I find the followings possible solutions:

• Solaris 10 zone, one can use set hostid property to set hostid
• There is Dtrace heck to set hostid
• Using Ldom in CMT system, the hostid is tie to MAC address that one can assign to ldom

Posted at 10:15PM Apr 16, 2008 by hstsao in General  |  Comments[0]

s9 containers

Sun just announce Solaris 9 containers with Solaris 10
Now Solaris 10 BrandZ support:

• Redhat 3 32-bit Containers
• Solaris 8 Containers
• Solaris 9 Containers

Key components for Solaris 9 Containers

• p2v conversion tool
• Solaris 9 Container environment
• Enterprise level support for Solaris 9 environments on Solaris 10

Requirements

• The Solaris 9 Containers subscription includes the right to use (RTU) the software and premium support for the Solaris 9 Containers software for a term of one year.
• Customers must have a valid Solaris 9 license and either a Solaris 10 premium subscription or a Sun System Gold or Platinum Service Plan to cover the support of the Solaris 10 OS
• Ultrasparc Server with Solaris 10 8/07 or later with additional Solaris 10 patch 127111-01 or later

Optional Service Enterprise Migration Implementation service

• Assessing that a customer application is suitable for Solaris 9 Container
• Building the Solaris 9 container for the application
• Tuning and testing of the application in the Solaris 9 Container

Posted at 09:50PM Apr 16, 2008 by hstsao in Solaris 10  |  Comments[0]