« September 2008
SunMonTueWedThuFriSat
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
    
       
Today

Blog::Navigation

Blog::Editing

Bookmarks::Blogroll

Blog::Referrers

Today's Page Hits: 136

Site notes

This page validates as XHTML 1.0, and will look much better in a browser that supports web standards, but it is accessible to any browser or Internet device. It was created using techniques detailed at glish.com/css/.

Powered by Roller Weblogger.
Main | Next page »
Monday Jul 28, 2008

sort in 7bit and 8bit

IHAC wrote to me that depend on which subnet he login to T5120 the timex sort file>file.out on /tmp give very different result;
one is about 8 sec
one is about 2. minute 50 sec
a factor of 21 difference.

Turnout that the locale is different in two way of login to the system one set LANG=C
one setto LNAG=en_US.utf8
the result for LANG=en_US is similar to LANG=C

Wednesday Jul 16, 2008

Flextronics IB switches

Flextronics provide some very inexpensive IB switches
infiniband for Mass Interesting article that talk about in expensive IB for HPC cluster

Tuesday Jul 15, 2008

3 socket AMD system

Base on today HT-1.0, it seems that 3 socket AMD 8xxx CPU with the 3 HT-links will be a very balance IO server
with 3 HT links, each CPU will have HT link to other CPU and one HT link to IO chipset.

Wednesday Jun 18, 2008

sb6048 IB-NEM switch connection without IB swicth

3x24 datacenter is perfect partner with sb6048 IB-NEM IB switch
What is the connection possibility for IB-NEM without a iB Switch?
Each IB-NEm contaions two 24 x4 port swicth chipset and provide dual connection to the 12 blades in one of the sb6048 chassis.
The other 2 12 port are presented as two set of 4 x12 port for outgoing connections.
The 12 blades in chassis are connected with fullly no blocking CLOS network.
betwenn the chassies the followings are possible:

  • 4 x12 cables between two chassis for 24 blades for non blocking CLOS
  • 2 x12 cables between 4 chassis for 48 blades for 50% blocking CLOS
  • 1 x12 cables between 5 chassis for 60 blades for 25% blocking CLOS

3x24 datacenter IB switch and sb6048

Sun recently introduce 3x24 Datacenter IB switch it is based on three 24 port x4 chipset
the key inovation is that it present the 72 x4 port as 24 x12 port in 1U form factor
It is best partner with the leaf switch IB-NEM in the Blade 6048
For smaller network (<=288) one could use 3x 24 datacenter swicth to build a 3 stages IB network.

  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require single 3x24 datacenter switch
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades require 2 3 x24 datacenter switches
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades require 4 3 x24 datacenter switches

Thursday May 01, 2008

x41450 and x4450 support of window 2008 server

The X4150 and X4450 are Certified to run Windows 2008 today(may/01/2008). There are no additional drivers needed for these platforms to work with Windows 2008. However, there is a minimum BIOS requirement
X4150 Mircosoft Windows 2008 Certification link: x4150 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:
X4450 Mircosoft Windows 2008 Certification link: x4450 Micosoft Window 2008 Certification link
Note: You will need BIOS version 1ADQW020 or greater to support Windows 2008. BIOS version 1ADQW020 can be found here:

rocks 5.0 released

download; http://www.rocksclusters.org/wordpress/?page_id=82 new features:

  • Xen Support:
    You can use the Xen Roll to create VM Containers: physical machines that are used to hold Xen-based virtual machines. The Rocks Command Line was expanded to help build and maintain VMs (e.g., rocks create host vm compute-0-0-0? is used to install a VM).
  • Fully-Programmable Partitioning
    The partitioning of client nodes (e.g., compute nodes and tile nodes) has been retooled. You can supply Red Hat partitioning directives to any node by writing a program in the pre section which populates the file /tmp/user_partitioning_info. The program can be as simple as small bash script that echos Red Hat partitioning directives or as complex as a python program that outputs partitioning info based on: the nodes name, the nodes membership, the number of disks in the node or the type of disks in the node. See the Base Roll documentation for details.
  • OS: Based on CentOS release 5/update 1 and all updates as of April 29, 2008
  • Condor: updated to v7.0.1
  • Ganglia: Ganglia Monitor Core updated to v3.0.7
  • Ganglia: phpsysinfo updated to v2.5.4
  • Ganglia: rrdtool updated to v1.2.23
  • HPC: The HPC roll is now optional. You can build a bare-bones cluster by using only the Kernel Roll, OS Rolls (disk 1 and 2) and the Base Roll
  • HPC: MPICH updated to v1.2.7 patch 1)
  • HPC: MPICH2 added to the roll (v1.0.6 patch 1)
  • HPC: OpenMPI added to the roll (v1.2.6)
  • SGE: SGE updated to 6.1 update 4
  • SGE: Added tight integration for SGE and OpenMPI
  • Area51: chkrootkit updated to v0.48
  • Area51: tripwire updated to v2.4.1.2
  • Bio: Biopython updated to v1.45
  • Bio: Clustalw updated to v2.0.5
  • Bio: Fasta updated to v35.3.5
  • Bio: NCBI toolbox updated to Mar 2008 version
  • Bio: MpiBlast updated to v1.5.0-pio and is patched against the NCBI toolbox Mar 2008 version
  • Bio: Phylip updated to v3.67
  • Bio: T_coffee updated to v5.65
  • Bio: Gromacs and MrBayes are now MPI Enabled and compiled against rocks-openmpi

Sunday Apr 27, 2008

IB network design leaf and core switch trade-off

Current IB switch based on Mellanox IS III 24 port 4x chipset support upto 288 in 3 stages and 3456 in 5 stages.
Recently Sun has build in leaf switch in the Blade 6048 rack.
If one use 3stages IB switches with sb6048 build in leaf switch one will create 5 stages network, and if one use 5 stages IB switch then we will create 7 stage network.
TACC support more than 3456 nodes so it need 2 3456 switches and it use IB-NEM to create 7 stage CLOS network
For smaller network (<=288) one could use many 24 port switches as core swicthes to build a 3 stages IB network.

  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade require 3x24 port core switches
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade require 4x24 port core switches
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades require 6 x24 port core switches
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades require 12 x24 port core switches
5 stages network using NEM IB leaf switch
  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch
3 stages network using IB PCI-E EM :
  • 6 chassises of 12 blades: 1 1/2 sb6048 = 72 blade and 72 port core switch
  • 8 chassises of 12 blades: 2 sb6048 = 96 blade and 96 port core switch
  • 12 chassises of 12 blades: 3 sb6048 = 144 blades 144 port core switch
  • 24 chassises of 12 blades: 6 sb6048 = 288 blades and 288 port core switch
For much larger network (>288) one could use many 288 port switches as core switches to build a 5 stages IB network. one can also use the dual port IB EM modules connect to core switches , for <=288 of 3 stages network and <=3465 for 5 stages networks.
Of course one use many core switches to construct larger IB network.
The same argument can apply to rack with rack server:
there are design tradeoff with big IB switch and has small leaf switches in each rack.
for examples if we use 36 1U server as building block and 3 24 port leaf switches in a rack , then there are two way to build IB network
  • use 3 stages core switches (<=288) to build a 5 stages network
  • use many 24 port switches as core switch for 3 stages network
The main difference between 3 stages and 5 stages IB network is the two switch hub difference: ~2x200 nanosecond difference for 24 port switch chipset.

mellanox InfiniScale IV architecture

Mellanox announced InfiniScale IV architecture in sc2007
the benefits are:

  • 40Gb/s server and storage interconnect (4x QDR)
  • 120Gb/s switch-to-switch interconnect (12x QDR)
  • 60 nanosecond switch hop latency
  • 36-port switch devices for optimal scalability
  • Adaptive Routing to optimize data traffic flow
  • Congestion control to avoid hot spot
Using 36 port silicon to build an non-blocking CLOS switches, in 3 layers setup, one can get(for 24 port chipset we can get 288 port in 3 stages):
  • 72 ports: 6 chips, 9 links between chips, 2 cores and 4 leafs
  • 108 ports: 9 chips, 6 links between chips, 3 cores and 6 leafs
  • 216 ports: 18 chips, 3 links between chips, 6 cores and 12 leafs
  • 324 ports: 27 chips, 2 links between chips, 9 cores and 18 leafs
  • 648 ports: 54 chips, 1 link between chips, 18 cores and 36 leafs
For 5 stage non-blocking CLOS network one can get (24 port chipset we can get to 3456 port in 5 stages)
  • 2 x 648 = 1296 port
  • 3 x 648 = 1944 port
  • 4 x 648 = 2592 port
  • 6 x 648 = 3888 port
  • 9x 648 = 5832 port
  • 12 x 648 = 7776 port
  • 18x 648 = 11664 port

as comparison with InfiniScale IV we list the InfiniScale III 24 port features here:
  • Twenty-four 10 or 20Gb/s Infiniband 4X ports or eight 30 or 60Gb/s Infiniband 12X ports (or any combination)
  • 480Gb/s (SDR version) or 960Gb/s (DDR version) of total switching bandwidt
  • Scalable to thousands of Ports
  • 96 integrated 2.5Gb/s (SDR version) or 5Gb/s (DDR version) SerDes interfaces (physical layer)
  • Auto-negotiation of port link speed
  • Ultra low latency cut-through switching (less than 200 nanoseconds) MTU Size from 256 to 2K bytes
  • Supports Multi-Protocol Applications for Clustering, Communication, and Storage
  • Integrated Subnet Management Agent (SMA)

Thursday Apr 17, 2008

Solaris Operating system Life Cycle

This import link describe the solaris OS life Cycle Sun's Life Cycle Model for the Solaris Operating System
Please note that market conditions and requirements can cause the exact date of the transition to vary
Key day to remember:

  • Solaris 9 launched in May 2002
  • Solaris 10 launched in January 2005

Wednesday Apr 16, 2008

x2200M2 QC U.S. Education Essentials Matching Grant

This year MG has very aggressive 60% discount on most products The following x2200 M2 QC is particular interesting for HPC customer

  • A85-FFZ2-H-8GB-JL8
  • Sun Fire X2200 M2 x64 Server: 2x Quad Core AMD Opteron Model 2354 processors (2.2GHz/95W, B3), 4x 2GB registered ECC DDR2-667 memory, no hard drives, no optical drive, 1x PSU, Service Processor, 4x 10/100/1000 Ethernet ports, 6x USB 2.0 ports, 1x I/O riser card with 2x PCI-Express x8 slots, no power cords, order Geo-specific x-option. RoHS-5. Standard Configuration.
  • list $2,905.00
  • MG $1,162.00

change hostid

Recently a customer request some way to migrate from older Sun server and solaris to newer server and Solaris but with one condition: need to preserve the hostid so the license server can use the license file with old hostid.
After some research, I find the followings possible solutions:

  • Solaris 10 zone, one can use set hostid property to set hostid
  • There is Dtrace heck to set hostid
  • Using Ldom in CMT system, the hostid is tie to MAC address that one can assign to ldom

s9 containers

Sun just announce Solaris 9 containers with Solaris 10
Now Solaris 10 BrandZ support:

  • Redhat 3 32-bit Containers
  • Solaris 8 Containers
  • Solaris 9 Containers
Key components for Solaris 9 Containers
  • p2v conversion tool
  • Solaris 9 Container environment
  • Enterprise level support for Solaris 9 environments on Solaris 10

Requirements
  • The Solaris 9 Containers subscription includes the right to use (RTU) the software and premium support for the Solaris 9 Containers software for a term of one year.
  • Customers must have a valid Solaris 9 license and either a Solaris 10 premium subscription or a Sun System Gold or Platinum Service Plan to cover the support of the Solaris 10 OS
  • Ultrasparc Server with Solaris 10 8/07 or later with additional Solaris 10 patch 127111-01 or later
Optional Service Enterprise Migration Implementation service
  • Assessing that a customer application is suitable for Solaris 9 Container
  • Building the Solaris 9 container for the application
  • Tuning and testing of the application in the Solaris 9 Container

s10u5 new features

  • Capping CPU resource usage
  • Support for Enhanced Intel SpeedStep and AMD PowerNow power management technology.
  • Virtual Network Computing (VNC) now included with Solaris
  • Infiniband flash update tool
  • Sockets Direct Protocol (SDP)
  • IP addressing ability for IBTF interfaces
  • SAS multipathing support
  • SunVTS(TM) 7.0
  • iSNS Client for iSCSI target
  • Persistent Group Reservation for iSCSI target
  • Hardware Key Generation (Crypto Framework feature)

Tuesday Jan 01, 2008

SMF and tcp tuning

Before Solaris 10 when one want to pre set some ndd setting, one setup some rc3 script e.g. S99ndd that contain ndd /dev/tcp scripts.
In solaris 10 there is similar but different way of setting these paramaters using the new SMF.Directory server TCP tuning I also copy these text in this page #

Edit the following xml file according to your environment and save the file as /var/svc/manifest/site/ndd-nettune.xml.

<?xml version="1.0"?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/ service_bundle.dtd.1">
<!--
    ident    "@(#)ndd-nettune.xml    1.0    04/09/21 SMI"
-->

<service_bundle type='manifest' name='SUNWcsr:ndd'>

<service
    name='network/ndd-nettune'
    type='service'
    version='1'>

			<create_default_instance enabled='true' />

			<single_instance />

		<dependency
        name='fs-minimal'
        type='service'
        grouping='require_all'
        restart_on='none'>

			<service_fmri value='svc:/system/filesystem/minimal' />
		</dependency>

		<dependency
        name='loopback-network'
        grouping='require_any'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/loopback' />
		</dependency>

		<dependency
        name='physical-network'
        grouping='optional_all'
        restart_on='none'
        type='service'>
			<service_fmri value='svc:/network/physical' />
		</dependency>

		<exec_method
        type='method'
        name='start'
        exec='/lib/svc/method/ndd-nettune'
        timeout_seconds='3' />
		</exec_method>

		<exec_method
               type='method'
               name='stop'
               exec=':true'
               timeout_seconds='3' >
		</exec_method>

		<property_group name='startd' type='framework'>
			<propval name='duration' type='astring'  value='transient' />
		</property_group>

		<stability value='Unstable' />

		<template>
			<common_name>
				<loctext xml:lang='C'>
					ndd network tuning

				</loctext>
			</common_name>
			<documentation>
				<manpage title='ndd' section='1M'
                manpath='/usr/share/man' />
			</documentation>
		</template>

</service>

</service_bundle>
  • Before you import the ndd-nettune.xml configuration, verify that the syntax is correct. You can do this by running the following command:


    $ svccfg validate /var/svc/manifest/site/ndd-nettune.xml
  • Import the configuration by running the following command:


    $ svccfg import /var/svc/manifest/site/ndd-nettune.xml

    For more information see the svccfg(1M) man page.

  • Copy the following shell script into /lib/svc/method/ndd-nettune.

    #!/sbin/sh
    #
    # ident   "@(#)ndd-nettune.xml    1.0     01/08/06 SMI"
    
    . /lib/svc/share/smf_include.sh
    . /lib/svc/share/net_include.sh
    
    # Make sure that the libraries essential to this stage of booting  can be found.
    LD_LIBRARY_PATH=/lib; export LD_LIBRARY_PATH
    echo "Performing Directory Server Tuning..." >> /tmp/smf.out
    /usr/sbin/ndd -set /dev/tcp tcp_conn_req_max_q 1024
    /usr/sbin/ndd -set /dev/tcp tcp_keepalive_interval 600000
    /usr/sbin/ndd -set /dev/tcp tcp_ip_abort_cinterval 10000
    /usr/sbin/ndd -set /dev/tcp tcp_ip_abort_interval 60000
    
    
    # Reset the library path now that we are past the critical stage
    unset LD_LIBRARY_PATH
  • Run svcadm to enable nettune (for more information, see the svcadm(1M) man page).

  • Run svcs -x (for more information see the svcs(1) man page).

  • one can download these files
    ndd-nettune.txt
    ndd-nettune.xml

    Copyright (C) 2003, hstsao