LaoTsao's Weblog (老曹的網路記)

sort in 7bit and 8bit

IHAC wrote to me that depend on which subnet he login to T5120 the timex sort file>file.out on /tmp give very different result;
one is about 8 sec
one is about 2. minute 50 sec
a factor of 21 difference.

Turnout that the locale is different in two way of login to the system one set LANG=C
one setto LNAG=en_US.utf8
the result for LANG=en_US is similar to LANG=C

Posted at 04:40AM Jul 28, 2008 by hstsao in Solaris 10  |  Comments[0]

Flextronics IB switches

Flextronics provide some very inexpensive IB switches
infiniband for Mass Interesting article that talk about in expensive IB for HPC cluster

Posted at 06:21AM Jul 16, 2008 by hstsao in HPTC  |  Comments[0]

3 socket AMD system

Base on today HT-1.0, it seems that 3 socket AMD 8xxx CPU with the 3 HT-links will be a very balance IO server
with 3 HT links, each CPU will have HT link to other CPU and one HT link to IO chipset.

Posted at 09:22AM Jul 15, 2008 by hstsao in HPTC  |  Comments[0]

sb6048 IB-NEM switch connection without IB swicth

3x24 datacenter is perfect partner with sb6048 IB-NEM IB switch
What is the connection possibility for IB-NEM without a iB Switch?
Each IB-NEm contaions two 24 x4 port swicth chipset and provide dual connection to the 12 blades in one of the sb6048 chassis.
The other 2 12 port are presented as two set of 4 x12 port for outgoing connections.
The 12 blades in chassis are connected with fullly no blocking CLOS network.
betwenn the chassies the followings are possible:

• 4 x12 cables between two chassis for 24 blades for non blocking CLOS
• 2 x12 cables between 4 chassis for 48 blades for 50% blocking CLOS
• 1 x12 cables between 5 chassis for 60 blades for 25% blocking CLOS

Posted at 10:45PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

3x24 datacenter IB switch and sb6048

Sun recently introduce 3x24 Datacenter IB switch it is based on three 24 port x4 chipset
the key inovation is that it present the 72 x4 port as 24 x12 port in 1U form factor
It is best partner with the leaf switch IB-NEM in the Blade 6048
For smaller network (<=288) one could use 3x 24 datacenter swicth to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require single 3x24 datacenter switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 2 3 x24 datacenter switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 4 3 x24 datacenter switches

Posted at 10:30PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

x41450 and x4450 support of window 2008 server

The X4150 and X4450 are Certified to run Windows 2008 today(may/01/2008). There are no additional drivers needed for these platforms to work with Windows 2008. However, there is a minimum BIOS requirement
X4150 Mircosoft Windows 2008 Certification link: x4150 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:
X4450 Mircosoft Windows 2008 Certification link: x4450 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:

Posted at 03:11PM May 01, 2008 by hstsao in General  |  Comments[0]

rocks 5.0 released

download; http://www.rocksclusters.org/wordpress/?page_id=82 new features:

• Xen Support:
  You can use the Xen Roll to create VM Containers: physical machines that are used to hold Xen-based virtual machines. The Rocks Command Line was expanded to help build and maintain VMs (e.g., rocks create host vm compute-0-0-0? is used to install a VM).
• Fully-Programmable Partitioning
  The partitioning of client nodes (e.g., compute nodes and tile nodes) has been retooled. You can supply Red Hat partitioning directives to any node by writing a program in the pre section which populates the file /tmp/user_partitioning_info. The program can be as simple as small bash script that echos Red Hat partitioning directives or as complex as a python program that outputs partitioning info based on: the nodes name, the nodes membership, the number of disks in the node or the type of disks in the node. See the Base Roll documentation for details.
• OS: Based on CentOS release 5/update 1 and all updates as of April 29, 2008
• Condor: updated to v7.0.1
• Ganglia: Ganglia Monitor Core updated to v3.0.7
• Ganglia: phpsysinfo updated to v2.5.4
• Ganglia: rrdtool updated to v1.2.23
• HPC: The HPC roll is now optional. You can build a bare-bones cluster by using only the Kernel Roll, OS Rolls (disk 1 and 2) and the Base Roll
• HPC: MPICH updated to v1.2.7 patch 1)
• HPC: MPICH2 added to the roll (v1.0.6 patch 1)
• HPC: OpenMPI added to the roll (v1.2.6)
• SGE: SGE updated to 6.1 update 4
• SGE: Added tight integration for SGE and OpenMPI
• Area51: chkrootkit updated to v0.48
• Area51: tripwire updated to v2.4.1.2
• Bio: Biopython updated to v1.45
• Bio: Clustalw updated to v2.0.5
• Bio: Fasta updated to v35.3.5
• Bio: NCBI toolbox updated to Mar 2008 version
• Bio: MpiBlast updated to v1.5.0-pio and is patched against the NCBI toolbox Mar 2008 version
• Bio: Phylip updated to v3.67
• Bio: T_coffee updated to v5.65
• Bio: Gromacs and MrBayes are now MPI Enabled and compiled against rocks-openmpi

Posted at 07:25AM May 01, 2008 by hstsao in HPTC  |  Comments[0]

IB network design leaf and core switch trade-off

Current IB switch based on Mellanox IS III 24 port 4x chipset support upto 288 in 3 stages and 3456 in 5 stages.
Recently Sun has build in leaf switch in the Blade 6048 rack.
If one use 3stages IB switches with sb6048 build in leaf switch one will create 5 stages network, and if one use 5 stages IB switch then we will create 7 stage network.
TACC support more than 3456 nodes so it need 2 3456 switches and it use IB-NEM to create 7 stage CLOS network
For smaller network (<=288) one could use many 24 port switches as core swicthes to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require 3x24 port core switches
• 8 chassises of 12 blades: 2 sb6048 = 96 blade require 4x24 port core switches
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 6 x24 port core switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 12 x24 port core switches

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

• use 3 stages core switches (<=288) to build a 5 stages network
• use many 24 port switches as core switch for 3 stages network

Posted at 07:36PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

mellanox InfiniScale IV architecture

Mellanox announced InfiniScale IV architecture in sc2007
the benefits are:

• 40Gb/s server and storage interconnect (4x QDR)
• 120Gb/s switch-to-switch interconnect (12x QDR)
• 60 nanosecond switch hop latency
• 36-port switch devices for optimal scalability
• Adaptive Routing to optimize data traffic flow
• Congestion control to avoid hot spot

• 72 ports: 6 chips, 9 links between chips, 2 cores and 4 leafs
• 108 ports: 9 chips, 6 links between chips, 3 cores and 6 leafs
• 216 ports: 18 chips, 3 links between chips, 6 cores and 12 leafs
• 324 ports: 27 chips, 2 links between chips, 9 cores and 18 leafs
• 648 ports: 54 chips, 1 link between chips, 18 cores and 36 leafs

• 2 x 648 = 1296 port
• 3 x 648 = 1944 port
• 4 x 648 = 2592 port
• 6 x 648 = 3888 port
• 9x 648 = 5832 port
• 12 x 648 = 7776 port
• 18x 648 = 11664 port

• Twenty-four 10 or 20Gb/s Infiniband 4X ports or eight 30 or 60Gb/s Infiniband 12X ports (or any combination)
• 480Gb/s (SDR version) or 960Gb/s (DDR version) of total switching bandwidt
• Scalable to thousands of Ports
• 96 integrated 2.5Gb/s (SDR version) or 5Gb/s (DDR version) SerDes interfaces (physical layer)
• Auto-negotiation of port link speed
• Ultra low latency cut-through switching (less than 200 nanoseconds) MTU Size from 256 to 2K bytes
• Supports Multi-Protocol Applications for Clustering, Communication, and Storage
• Integrated Subnet Management Agent (SMA)

Posted at 05:38PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

Solaris Operating system Life Cycle

This import link describe the solaris OS life Cycle Sun's Life Cycle Model for the Solaris Operating System
Please note that market conditions and requirements can cause the exact date of the transition to vary
Key day to remember:

• Solaris 9 launched in May 2002
• Solaris 10 launched in January 2005

Posted at 05:54AM Apr 17, 2008 by hstsao in Solaris 10  |  Comments[0]

x2200M2 QC U.S. Education Essentials Matching Grant

This year MG has very aggressive 60% discount on most products The following x2200 M2 QC is particular interesting for HPC customer

• A85-FFZ2-H-8GB-JL8
• Sun Fire X2200 M2 x64 Server: 2x Quad Core AMD Opteron Model 2354 processors (2.2GHz/95W, B3), 4x 2GB registered ECC DDR2-667 memory, no hard drives, no optical drive, 1x PSU, Service Processor, 4x 10/100/1000 Ethernet ports, 6x USB 2.0 ports, 1x I/O riser card with 2x PCI-Express x8 slots, no power cords, order Geo-specific x-option. RoHS-5. Standard Configuration.
• list $2,905.00
• MG $1,162.00

Posted at 10:27PM Apr 16, 2008 by hstsao in HPTC  |  Comments[0]

change hostid

Recently a customer request some way to migrate from older Sun server and solaris to newer server and Solaris but with one condition: need to preserve the hostid so the license server can use the license file with old hostid.
After some research, I find the followings possible solutions:

• Solaris 10 zone, one can use set hostid property to set hostid
• There is Dtrace heck to set hostid
• Using Ldom in CMT system, the hostid is tie to MAC address that one can assign to ldom

Posted at 10:15PM Apr 16, 2008 by hstsao in General  |  Comments[0]

s9 containers

Sun just announce Solaris 9 containers with Solaris 10
Now Solaris 10 BrandZ support:

• Redhat 3 32-bit Containers
• Solaris 8 Containers
• Solaris 9 Containers

• p2v conversion tool
• Solaris 9 Container environment
• Enterprise level support for Solaris 9 environments on Solaris 10

• The Solaris 9 Containers subscription includes the right to use (RTU) the software and premium support for the Solaris 9 Containers software for a term of one year.
• Customers must have a valid Solaris 9 license and either a Solaris 10 premium subscription or a Sun System Gold or Platinum Service Plan to cover the support of the Solaris 10 OS
• Ultrasparc Server with Solaris 10 8/07 or later with additional Solaris 10 patch 127111-01 or later

• Assessing that a customer application is suitable for Solaris 9 Container
• Building the Solaris 9 container for the application
• Tuning and testing of the application in the Solaris 9 Container

Posted at 09:50PM Apr 16, 2008 by hstsao in Solaris 10  |  Comments[0]

s10u5 new features

• Capping CPU resource usage
• Support for Enhanced Intel SpeedStep and AMD PowerNow power management technology.
• Virtual Network Computing (VNC) now included with Solaris
• Infiniband flash update tool
• Sockets Direct Protocol (SDP)
• IP addressing ability for IBTF interfaces
• SAS multipathing support
• SunVTS(TM) 7.0
• iSNS Client for iSCSI target
• Persistent Group Reservation for iSCSI target
• Hardware Key Generation (Crypto Framework feature)

Posted at 09:28PM Apr 16, 2008 by hstsao in Solaris 10  |  Comments[0]

SMF and tcp tuning

Before Solaris 10 when one want to pre set some ndd setting, one setup some rc3 script e.g. S99ndd that contain ndd /dev/tcp scripts.
In solaris 10 there is similar but different way of setting these paramaters using the new SMF.Directory server TCP tuning I also copy these text in this page #

Edit the following xml file according to your environment and save the file as /var/svc/manifest/site/ndd-nettune.xml.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/ service_bundle.dtd.1">
<!--
    ident    "@(#)ndd-nettune.xml    1.0    04/09/21 SMI"
-->

<service_bundle type='manifest' name='SUNWcsr:ndd'>

<service
    name='network/ndd-nettune'
    type='service'
    version='1'>

			<create_default_instance enabled='true' />

			<single_instance />

		<dependency
        name='fs-minimal'
        type='service'
        grouping='require_all'
        restart_on='none'>

			<service_fmri value='svc:/system/filesystem/minimal' />
		</dependency>

		<dependency
        name='loopback-network'
        grouping='require_any'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/loopback' />
		</dependency>

		<dependency
        name='physical-network'
        grouping='optional_all'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/physical' />
		</dependency>

		<exec_method
        type='method'
        name='start'
        exec='/lib/svc/method/ndd-nettune'
        timeout_seconds='3' />
		</exec_method>

		<exec_method
               type='method'
               name='stop'
               exec=':true'
               timeout_seconds='3' >
		</exec_method>

		<property_group name='startd' type='framework'>
			<propval name='duration' type='astring'  value='transient' />
		</property_group>

		<stability value='Unstable' />

		<template>
			<common_name>
				<loctext xml:lang='C'>
					ndd network tuning

				</loctext>
			</common_name>
			<documentation>
				<manpage title='ndd' section='1M'
                manpath='/usr/share/man' />
			</documentation>
		</template>

</service>

</service_bundle>

$ svccfg validate /var/svc/manifest/site/ndd-nettune.xml

$ svccfg import /var/svc/manifest/site/ndd-nettune.xml

#!/sbin/sh
#
# ident   "@(#)ndd-nettune.xml    1.0     01/08/06 SMI"

. /lib/svc/share/smf_include.sh
. /lib/svc/share/net_include.sh

# Make sure that the libraries essential to this stage of booting  can be found.
LD_LIBRARY_PATH=/lib; export LD_LIBRARY_PATH
echo "Performing Directory Server Tuning..." >> /tmp/smf.out
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 1024
/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 600000
/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_interval 60000


# Reset the library path now that we are past the critical stage
unset LD_LIBRARY_PATH

Posted at 11:42AM Jan 01, 2008 by hstsao in Solaris 10  |  Comments[0]