nicky veitch's Weblog nicky veitch's Weblog

Solaris 10 introduced the nemo framework for drivers and Solaris Nevada has more projects which build on said framework. In Update 2 of Solaris 10 support for data link aggregation was added which means we can build fat network pipes from most nics :) without any trunking software.

From the manual for dladm

     The dladm command is used to configure data-links. A config-
     ured  data-link  is  represented  in the system as a STREAMS
     DLPI (v2) interface which  may  be  plumbed  under  protocol
     stacks  such  as  TCP/IP.  Each data-link relies on either a
     single network device or an aggregation of devices  to  send
     packets to or receive packets from a network.

Heres it on a galaxy but you can do the same on a t2000 using the new e1000g driver.

oaf316# ifconfig -a unplumb
oaf316# dladm show-dev

e1000g0         link: up        speed: 1000  Mbps       duplex: full
e1000g1         link: up        speed: 1000  Mbps       duplex: full
e1000g2         link: up        speed: 1000  Mbps       duplex: full
e1000g3         link: up        speed: 1000  Mbps       duplex: full

Now we know what devices are available for our aggregation. We will make a an aggregation of 2 of the nics.

 
oaf316# dladm create-aggr -d e1000g0 -d e1000g3 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)
           device       address                 speed           duplex  link    state
           e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      standby
           e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      standby

Agrregation completes and the link is in standby mode, next we need to plumb it.

oaf316# ifconfig aggr1 plumb
oaf316# ifconfig aggr1 10.1.10.1 netmask 255.255.255.0 up

regular ifconfig to setup the link. Lets check the device state now.

dladm show-aggr 1
key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)
           device       address                 speed           duplex  link    state
           e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached
           e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached

We add some more nics to the device while the device is up and running.

oaf316# dladm add-aggr -d e1000g1 -d e1000g2 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)
           device       address                 speed           duplex  link    state
           e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached
           e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           e1000g1      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           e1000g2      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

Now lets show off and remove a nic from the link

oaf316# dladm remove-aggr -d e1000g0 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b3 (auto)
           device       address                 speed           duplex  link    state
           e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached
           e1000g1      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached
           e1000g2      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

heres the postinstall script which we use with some of our systems, you will have to change the IP.

# script to setup link aggregation on nics which are not in use
# it trys to ignore unsupported nics.
# This script can be used as part of a jumpstart.
#
# man dladm

PRE=/
[ -f /a/usr/sbin/dladm ] && PRE=/a

PATH=$PRE/usr/bin:$PRE/usr/sbin
export PATH

# ip for configured device to use  eg 10.1.1.1
IP=10.1.1.1

# netmasks for configured device eg 255.255.255.0
Netmasks=255.255.255.0

# set this to "e1000g0 nge0 bge1" etc this can be left blank and we try to use
# other gld nics
NicsToUse=



showError() {
        echo "$0: $1"
        exit 1
}

# exit if no ip or netmask
[ -z "$IP" -o  -z "$Netmasks" ] && showError "IP and Netmasks must be defined"

# check network devices exists
if [ -z "$NicsToUse" ]; then
        ifconfig -a plumb 2>/dev/null
        NicsToUse=`ifconfig -a |awk -F: '//dev/null
        [ `dladm show-link $nic |grep -v -c legacy` ] && vNics="$vNics -d $nic"
done

# no nics supported by dladm
[ -z "$vNics" ] && showError "No supported nics on system"

# configure and plumb device
dladm create-aggr -R $PRE $vNics 1
[ $? != 0 ] && showError "error configuring aggr1 with dladm and $vNics"

ifconfig aggr1 plumb
ifconfig aggr1 $IP netmask $Netmasks up
[ $? != 0 ] && showError "error bringing up aggr1 with ifconfig $IP netmask $Netmasks "

# store nic details
echo $IP >$PRE/etc/hostname.aggr1
IP=`echo $IP|cut -f1-3 -d\.`.0
echo "$IP $Netmasks" >>$PRE/etc/netmasks

You will also want to increase the number of soft rings used by your aggregations. This can be done via /etc/system or via mdb as the default is 2 per interface.

oaf316# mdb -kw 
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 ufs ip sctp usba fcp 
fctl nca random md lofs zfs nfs sppp crypto cpc fcip logindmux ptm ]
> ip_soft_rings_cnt/W 8
ip_soft_rings_cnt:              0x2             =       0x8
> $q

This increases the number of soft rings from 2 to 8. If your aggregation is already plumbed you will need to replumb it to take advantage of the extra rings. To make this permanent you will need to add it to /etc/system

set ip:ip_soft_rings_cnt=8

Be warned this will do it for each link after the next reboot. More to come on t2000 and link aggregation.