LaoTsao's Weblog (老曹的網路記)

sb6048 IB-NEM switch connection without IB swicth

3x24 datacenter is perfect partner with sb6048 IB-NEM IB switch
What is the connection possibility for IB-NEM without a iB Switch?
Each IB-NEm contaions two 24 x4 port swicth chipset and provide dual connection to the 12 blades in one of the sb6048 chassis.
The other 2 12 port are presented as two set of 4 x12 port for outgoing connections.
The 12 blades in chassis are connected with fullly no blocking CLOS network.
betwenn the chassies the followings are possible:

• 4 x12 cables between two chassis for 24 blades for non blocking CLOS
• 2 x12 cables between 4 chassis for 48 blades for 50% blocking CLOS
• 1 x12 cables between 5 chassis for 60 blades for 25% blocking CLOS

Posted at 10:45PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

3x24 datacenter IB switch and sb6048

Sun recently introduce 3x24 Datacenter IB switch it is based on three 24 port x4 chipset
the key inovation is that it present the 72 x4 port as 24 x12 port in 1U form factor
It is best partner with the leaf switch IB-NEM in the Blade 6048
For smaller network (<=288) one could use 3x 24 datacenter swicth to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require single 3x24 datacenter switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 2 3 x24 datacenter switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 4 3 x24 datacenter switches

Posted at 10:30PM Jun 18, 2008 by hstsao in HPTC  |  Comments[0]

x41450 and x4450 support of window 2008 server

The X4150 and X4450 are Certified to run Windows 2008 today(may/01/2008). There are no additional drivers needed for these platforms to work with Windows 2008. However, there is a minimum BIOS requirement
X4150 Mircosoft Windows 2008 Certification link: x4150 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:
X4450 Mircosoft Windows 2008 Certification link: x4450 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:

Posted at 03:11PM May 01, 2008 by hstsao in General  |  Comments[0]

rocks 5.0 released

download; http://www.rocksclusters.org/wordpress/?page_id=82 new features:

• Xen Support:
  You can use the Xen Roll to create VM Containers: physical machines that are used to hold Xen-based virtual machines. The Rocks Command Line was expanded to help build and maintain VMs (e.g., rocks create host vm compute-0-0-0? is used to install a VM).
• Fully-Programmable Partitioning
  The partitioning of client nodes (e.g., compute nodes and tile nodes) has been retooled. You can supply Red Hat partitioning directives to any node by writing a program in the pre section which populates the file /tmp/user_partitioning_info. The program can be as simple as small bash script that echos Red Hat partitioning directives or as complex as a python program that outputs partitioning info based on: the nodes name, the nodes membership, the number of disks in the node or the type of disks in the node. See the Base Roll documentation for details.
• OS: Based on CentOS release 5/update 1 and all updates as of April 29, 2008
• Condor: updated to v7.0.1
• Ganglia: Ganglia Monitor Core updated to v3.0.7
• Ganglia: phpsysinfo updated to v2.5.4
• Ganglia: rrdtool updated to v1.2.23
• HPC: The HPC roll is now optional. You can build a bare-bones cluster by using only the Kernel Roll, OS Rolls (disk 1 and 2) and the Base Roll
• HPC: MPICH updated to v1.2.7 patch 1)
• HPC: MPICH2 added to the roll (v1.0.6 patch 1)
• HPC: OpenMPI added to the roll (v1.2.6)
• SGE: SGE updated to 6.1 update 4
• SGE: Added tight integration for SGE and OpenMPI
• Area51: chkrootkit updated to v0.48
• Area51: tripwire updated to v2.4.1.2
• Bio: Biopython updated to v1.45
• Bio: Clustalw updated to v2.0.5
• Bio: Fasta updated to v35.3.5
• Bio: NCBI toolbox updated to Mar 2008 version
• Bio: MpiBlast updated to v1.5.0-pio and is patched against the NCBI toolbox Mar 2008 version
• Bio: Phylip updated to v3.67
• Bio: T_coffee updated to v5.65
• Bio: Gromacs and MrBayes are now MPI Enabled and compiled against rocks-openmpi

Posted at 07:25AM May 01, 2008 by hstsao in HPTC  |  Comments[0]

IB network design leaf and core switch trade-off

Current IB switch based on Mellanox IS III 24 port 4x chipset support upto 288 in 3 stages and 3456 in 5 stages.
Recently Sun has build in leaf switch in the Blade 6048 rack.
If one use 3stages IB switches with sb6048 build in leaf switch one will create 5 stages network, and if one use 5 stages IB switch then we will create 7 stage network.
TACC support more than 3456 nodes so it need 2 3456 switches and it use IB-NEM to create 7 stage CLOS network
For smaller network (<=288) one could use many 24 port switches as core swicthes to build a 3 stages IB network.

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require 3x24 port core switches
• 8 chassises of 12 blades: 2 sb6048 = 96 blade require 4x24 port core switches
• 12 chassises of 12 blades: 3 sb6048 = 144 blades require 6 x24 port core switches
• 24 chassises of 12 blades: 6 sb6048 = 288 blades require 12 x24 port core switches

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

• 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
• 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
• 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
• 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch

• use 3 stages core switches (<=288) to build a 5 stages network
• use many 24 port switches as core switch for 3 stages network

Posted at 07:36PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

mellanox InfiniScale IV architecture

Mellanox announced InfiniScale IV architecture in sc2007
the benefits are:

• 40Gb/s server and storage interconnect (4x QDR)
• 120Gb/s switch-to-switch interconnect (12x QDR)
• 60 nanosecond switch hop latency
• 36-port switch devices for optimal scalability
• Adaptive Routing to optimize data traffic flow
• Congestion control to avoid hot spot

• 72 ports: 6 chips, 9 links between chips, 2 cores and 4 leafs
• 108 ports: 9 chips, 6 links between chips, 3 cores and 6 leafs
• 216 ports: 18 chips, 3 links between chips, 6 cores and 12 leafs
• 324 ports: 27 chips, 2 links between chips, 9 cores and 18 leafs
• 648 ports: 54 chips, 1 link between chips, 18 cores and 36 leafs

• 2 x 648 = 1296 port
• 3 x 648 = 1944 port
• 4 x 648 = 2592 port
• 6 x 648 = 3888 port
• 9x 648 = 5832 port
• 12 x 648 = 7776 port
• 18x 648 = 11664 port

• Twenty-four 10 or 20Gb/s Infiniband 4X ports or eight 30 or 60Gb/s Infiniband 12X ports (or any combination)
• 480Gb/s (SDR version) or 960Gb/s (DDR version) of total switching bandwidt
• Scalable to thousands of Ports
• 96 integrated 2.5Gb/s (SDR version) or 5Gb/s (DDR version) SerDes interfaces (physical layer)
• Auto-negotiation of port link speed
• Ultra low latency cut-through switching (less than 200 nanoseconds) MTU Size from 256 to 2K bytes
• Supports Multi-Protocol Applications for Clustering, Communication, and Storage
• Integrated Subnet Management Agent (SMA)

Posted at 05:38PM Apr 27, 2008 by hstsao in HPTC  |  Comments[0]

Solaris Operating system Life Cycle

This import link describe the solaris OS life Cycle Sun's Life Cycle Model for the Solaris Operating System
Please note that market conditions and requirements can cause the exact date of the transition to vary
Key day to remember:

• Solaris 9 launched in May 2002
• Solaris 10 launched in January 2005

Posted at 05:54AM Apr 17, 2008 by hstsao in Solaris 10  |  Comments[0]

x2200M2 QC U.S. Education Essentials Matching Grant

This year MG has very aggressive 60% discount on most products The following x2200 M2 QC is particular interesting for HPC customer

• A85-FFZ2-H-8GB-JL8
• Sun Fire X2200 M2 x64 Server: 2x Quad Core AMD Opteron Model 2354 processors (2.2GHz/95W, B3), 4x 2GB registered ECC DDR2-667 memory, no hard drives, no optical drive, 1x PSU, Service Processor, 4x 10/100/1000 Ethernet ports, 6x USB 2.0 ports, 1x I/O riser card with 2x PCI-Express x8 slots, no power cords, order Geo-specific x-option. RoHS-5. Standard Configuration.
• list $2,905.00
• MG $1,162.00

Posted at 10:27PM Apr 16, 2008 by hstsao in HPTC  |  Comments[0]

change hostid

Recently a customer request some way to migrate from older Sun server and solaris to newer server and Solaris but with one condition: need to preserve the hostid so the license server can use the license file with old hostid.
After some research, I find the followings possible solutions:

• Solaris 10 zone, one can use set hostid property to set hostid
• There is Dtrace heck to set hostid
• Using Ldom in CMT system, the hostid is tie to MAC address that one can assign to ldom

Posted at 10:15PM Apr 16, 2008 by hstsao in General  |  Comments[0]

s9 containers

Sun just announce Solaris 9 containers with Solaris 10
Now Solaris 10 BrandZ support:

• Redhat 3 32-bit Containers
• Solaris 8 Containers
• Solaris 9 Containers

• p2v conversion tool
• Solaris 9 Container environment
• Enterprise level support for Solaris 9 environments on Solaris 10

• The Solaris 9 Containers subscription includes the right to use (RTU) the software and premium support for the Solaris 9 Containers software for a term of one year.
• Customers must have a valid Solaris 9 license and either a Solaris 10 premium subscription or a Sun System Gold or Platinum Service Plan to cover the support of the Solaris 10 OS
• Ultrasparc Server with Solaris 10 8/07 or later with additional Solaris 10 patch 127111-01 or later

• Assessing that a customer application is suitable for Solaris 9 Container
• Building the Solaris 9 container for the application
• Tuning and testing of the application in the Solaris 9 Container

Posted at 09:50PM Apr 16, 2008 by hstsao in Solaris 10  |  Comments[0]

s10u5 new features

• Capping CPU resource usage
• Support for Enhanced Intel SpeedStep and AMD PowerNow power management technology.
• Virtual Network Computing (VNC) now included with Solaris
• Infiniband flash update tool
• Sockets Direct Protocol (SDP)
• IP addressing ability for IBTF interfaces
• SAS multipathing support
• SunVTS(TM) 7.0
• iSNS Client for iSCSI target
• Persistent Group Reservation for iSCSI target
• Hardware Key Generation (Crypto Framework feature)

Posted at 09:28PM Apr 16, 2008 by hstsao in Solaris 10  |  Comments[0]

SMF and tcp tuning

Before Solaris 10 when one want to pre set some ndd setting, one setup some rc3 script e.g. S99ndd that contain ndd /dev/tcp scripts.
In solaris 10 there is similar but different way of setting these paramaters using the new SMF.Directory server TCP tuning I also copy these text in this page #

Edit the following xml file according to your environment and save the file as /var/svc/manifest/site/ndd-nettune.xml.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/ service_bundle.dtd.1">
<!--
    ident    "@(#)ndd-nettune.xml    1.0    04/09/21 SMI"
-->

<service_bundle type='manifest' name='SUNWcsr:ndd'>

<service
    name='network/ndd-nettune'
    type='service'
    version='1'>

			<create_default_instance enabled='true' />

			<single_instance />

		<dependency
        name='fs-minimal'
        type='service'
        grouping='require_all'
        restart_on='none'>

			<service_fmri value='svc:/system/filesystem/minimal' />
		</dependency>

		<dependency
        name='loopback-network'
        grouping='require_any'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/loopback' />
		</dependency>

		<dependency
        name='physical-network'
        grouping='optional_all'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/physical' />
		</dependency>

		<exec_method
        type='method'
        name='start'
        exec='/lib/svc/method/ndd-nettune'
        timeout_seconds='3' />
		</exec_method>

		<exec_method
               type='method'
               name='stop'
               exec=':true'
               timeout_seconds='3' >
		</exec_method>

		<property_group name='startd' type='framework'>
			<propval name='duration' type='astring'  value='transient' />
		</property_group>

		<stability value='Unstable' />

		<template>
			<common_name>
				<loctext xml:lang='C'>
					ndd network tuning

				</loctext>
			</common_name>
			<documentation>
				<manpage title='ndd' section='1M'
                manpath='/usr/share/man' />
			</documentation>
		</template>

</service>

</service_bundle>

$ svccfg validate /var/svc/manifest/site/ndd-nettune.xml

$ svccfg import /var/svc/manifest/site/ndd-nettune.xml

#!/sbin/sh
#
# ident   "@(#)ndd-nettune.xml    1.0     01/08/06 SMI"

. /lib/svc/share/smf_include.sh
. /lib/svc/share/net_include.sh

# Make sure that the libraries essential to this stage of booting  can be found.
LD_LIBRARY_PATH=/lib; export LD_LIBRARY_PATH
echo "Performing Directory Server Tuning..." >> /tmp/smf.out
/usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 1024
/usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 600000
/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
/usr/sbin/ndd -set /dev/tcp tcp_ip_abort_interval 60000


# Reset the library path now that we are past the critical stage
unset LD_LIBRARY_PATH

Posted at 11:42AM Jan 01, 2008 by hstsao in Solaris 10  |  Comments[0]

why blade?

In a very short time span Sun has introdce SB8000, SB8000P , SB6000 and SB6048 chassises and x8400, X8420 and X8440 for SB8000 and SB8000P and X6220, X6250, T6300,T6320 for SB6000 and SB6048.
One should ask what is the advantage of the blade?

• Serviceability: CMM,PCIE EM are hot pluggabke,NEM is cold replaceable without the need to open any bldes
• Reduce Cabling: with CMM's build in management network one use one cable to manage 10 or 12 ILOM. with shared power supply, one can use few power cable to support 10 or 12 blade servers
• Reduce Power requirement: due to the consolidated fan, overall power requirement is less than to corresponding 10 or 12 rack servers:
• IB leaf switch NEM for SB6048: reduce IB cabling with new IPASS+ connector and cables
• SB6000 support intel, AMD and UltraSPARC T1 and T2 chips sets

Posted at 08:05AM Jan 01, 2008 by hstsao in HPTC  |  Comments[0]

OpenSolaris Developer Preview on USB

OpenSolaris Developer Preview

I download the ISO and burn to a few CDs just in time for NJEdge.net annual event on Nov 01

I demonstrated the preview to few customer and even give out one of the CD to SE from cisco

Everyone was impressed

When I try to install on my 2GB USB drive, it complian that it need to be at least 4GB

Today I follow the instruction on this link

http://blogs.sun.com/dminer/entry/opensolaris_developer_preview_on_usb

Put the image on a 1GB USB stick the followings are some screen shot,

Keep in mind that the machine has nv b76 installed so the mercuial is already installed

bash-3.2# hg clone ssh://anon@hg.opensolaris.org/hg/caiman/distro_constructor

The authenticity of host 'hg.opensolaris.org (72.5.123.5)' can't be established.

RSA key fingerprint is 63:b9:7a:f3:dc:a4:47:a7:a5:50:ef:05:78:39:8d:ad.

Are you sure you want to continue connecting (yes/no)? yes

remote: Warning: Permanently added 'hg.opensolaris.org,72.5.123.5' (RSA) to the list of known hosts.

destination directory: distro_constructor

requesting all changes

adding changesets

adding manifests

adding file changes

added 46 changesets with 255 changes to 190 files

179 files updated, 0 files merged, 0 files removed, 0 files unresolved

bash-3.2# pwd

/export/home/htsao

bash-3.2# ls -lt |more

total 28196932

drwxr-xr-x 8 root root 512 Nov 2 09:55 distro_constructor

bash-3.2# cd distro_constructor

bash-3.2# ls

README packages src test_data tools usr

bash-3.2# cd tools

bash-3.2# ls

extract_postrun proc_slist.pl usbcopy

listusb proc_tracedata usbgen

bash-3.2#

bash-3.2# ./usbgen /export/home/htsao/in-preview.iso /export/home/htsao/usb-in-previ ew.iso `pwd` /tmp

/dev/rlofi/3: Unable to find Media type. Proceeding with system determined parameter s.

/dev/rlofi/3: 1433400 sectors in 2389 cylinders of 1 tracks, 600 sectors 699.9MB in 150 cyl groups (16 c/g, 4.69MB/g, 2240 i/g)

super-block backups (for fsck -F ufs -o b=#) at:

32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,

1344032, 1353632, 1363232, 1372832, 1382432, 1392032, 1401632, 1411232, 1420832, 1430432

Copying ISO contents to USB image

..................................................

..................................................

..................................................

..................................................

..................................................

..................................................

1289920 blocks

Creating temporary microroot

Replacing USB specific files

Copying microroot to USB

Cleaning up.

=== ./usbgen completed at Fri Nov 2 12:38:00 EDT 2

bash-3.2# ./usbcopy /export/home/ht sao/usb-in-preview.iso

Found the following USB devices:

0: /dev/rdsk/c0t0d0p0 984.0 MB PNY USB 2.0 FD PMAP

Enter the number of your choice: 0

WARNING: All data on your USB storage will be lost.

Are you sure you want to install to

PNY USB 2.0 FD PMAP, 984 MB at /dev/rdsk/c0t0d0p0 ?(y/n) y

umount: warning: /dev/dsk/c0t0d0s0 not in mnttab

umount: /dev/dsk/c0t0d0s0 not mounted

Copying image to USB device

44800+0 records in

44800+0 records out

real 16m47.259s

user 0m0.189s

sys 0m3.242s

Installing grub to USB device /dev/rdsk/c0t0d0s0

Completed copy to USB

bash-3.2#

Posted at 01:24PM Nov 02, 2007 by hstsao in opensolaris  | 

sun studio 12 installation experience

Recently I need to help a customer to test some code, and he is will to try it on Solaris

After the Solaris 08/07 installed on the new servers, one need to install the latest Sun Studio 12

As usually I search for the patches info and find this pages Studio 12 patches

I busy download all these patches for solaris 10

I also download the latest Sun Studio 12 from download Use the simple commands batch_installer --accept_sla It finish in no time

I then try to install the patches and find that most patches are already installed and I also find a statement on this pages Sun Studio 12 ML editions is the Sun studio 12 plus patches sunstudio

Sun Studio 12 Multi-Language Release Now Available! Sun Studio 12 in English, Japanese and Simplified Chinese, with IDE enhancements and bundled product patches!

I wish that it has a link tell me which patches was part of the releases

I just need to install the following patches

• 126498-02
• 126496-02
• 124873-02
• 127112-01

Posted at 07:40PM Oct 19, 2007 by hstsao in Sun Studio  |  Comments[1]