This week's tip provides a couple of options with respect to configuring your network interfaces.

The two ends of the network interface configuration spectrum are -

"Link aggregation, or IEEE 802.3ad, is a computer networking term which describes using multiple Ethernet network cables/ports in parallel to increase the link speed beyond the limits of any one single cable or port, and to increase the redundancy for higher availability." (ref: http://en.wikipedia.org/wiki/Link_aggregation)

    - and -

Logical interfaces allow you to assign multiple IP addresses to a single physical interface, and share the bandwidth of a single interface across multiple IP addresses.

Link Aggregation:
Over the years there have been a couple ways to configure link aggregation on Solaris, the current method is via the 'dladm' command. Here's a script to detect all physical interfaces on your host and create a single aggregated link assigned the name 'aggr1'.

    #!/bin/sh

    if [ $# -lt 1 ]; then
      echo "Usage: ./trunk.sh [ip address]"
      exit 1
    fi

    interfaces=`dladm show-link | grep -v LINK | awk '{print $1}'`
    dladm_cmd="dladm create-aggr -l passive"
    for i in $interfaces; do
      dladm_cmd="${dladm_cmd} -d $i"
    done
    dladm_cmd="${dladm_cmd} 1"

    echo "
    To enable trunking, execute the following commands:

    ${dladm_cmd}
    ifconfig aggr1 plumb ${1}/24 up
    dladm show-aggr
    echo \"${1}\" > /etc/hostname.aggr1
    svcadm disable network/physical:nwam
    svcadm enable  network/physical:default
    "
    exit 0

Examples of dladm command output used in the script:
    # dladm show-link
    LINK        CLASS    MTU    STATE    OVER
    e1000g4     phys     1500   up       --
    e1000g0     phys     1500   up       --
    e1000g2     phys     1500   unknown  --
    e1000g1     phys     1500   up       --
    e1000g3     phys     1500   unknown  --
    e1000g5     phys     1500   up       --

Create link aggregation named 'aggr1', using interfaces e1000g1, e1000g4 and e1000g5:
    # dladm create-aggr -l passive -d e1000g1 -d e1000g4 -d e1000g5 1

Show status of aggr1 interface:
    # dladm show-aggr
    LINK            POLICY   ADDRPOLICY           LACPACTIVITY  LACPTIMER   FLAGS
    aggr1           L4       auto                 passive       short       -----

Note that to use this feature you must coordinate with you network switch administrator. The policy and and aggregated interfaces must configured identically on the other end of the ethernet cables ...

We use the Link Aggregation Control Protocol (LACP) in passive mode to control simultaneous transmission on multiple interfaces. Any single stream is transmitted completely on an individual interface, but multiple simultaneous streams can be active across all interfaces.

Logical Interfaces:
A single physical interface can be assigned multiple IP addresses - you may have already seen this in my earlier post on zones, where I configured a set of zones to share a single physical network interface.

Use /network/physical:default service to manage network interfaces:
    # svcadm disable svc:/network/physical:nwam
    # svcadm enable  svc:/network/physical:default

Example: Configure a second address on interface bge0
    # ifconfig bge0
    bge0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
    inet 192.168.1.200 netmask ffffff00 broadcast 192.168.1.255

    # ifconfig bge0:1 plumb
    # ifconfig bge0:1 172.0.1.111/24 up
    # ifconfig bge0:1
    bge0:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
    inet 172.0.1.111 netmask ffffff00 broadcast 172.0.255.255

That was simple ! Your bge0 interface now participates on two subnets: 192.168.1.0/24 and 172.0.1.0/24.

To make this bge0:1 configuration permanent, create the file /etc/hostname.bge0:1 containing the ip address 172.0.1.111. Your logical interface will be initialized each time you boot your system.

Comments:

Jay,

Can Solaris handle 802.3ad *across* separate switches ? I know there are some proprietary solutions out there that have taken 802.3ad to this capability. Curious if Solaris can handle this.

Posted by Sean on September 05, 2008 at 12:29 PM EDT #

Hi Sean,
Solaris doesn't really care .... but in general the switch hardware I am familiar with does not support links across switches. Your requirement for availability/uptime can be met with ipmp on Solaris - see http://docs.sun.com/app/docs/doc/816-4554/enfoy?l=en&a=view&q=ipmp
HTH

Posted by jay on September 16, 2008 at 12:48 PM EDT #

The only real requirement for 3ad across multiple switches is that both switches are joined as one logical unit. Usually this is by means of a vendor proprietary uplink between the two units. I don't know what names other vendors use, but 3com's is called XRN Fabric. The OS of choice then knows no different.

Posted by Dan C on September 28, 2008 at 10:53 AM EDT #

Amazing page, simple and useful.

Wonder what are the requirement from the switch side. Is this LACP Cisco specific ?

What is the different of this technology with IPMP ?

Posted by fabio on October 17, 2008 at 01:42 AM EDT #

review

Posted by samantha on December 01, 2008 at 11:26 AM EST #

Jay, if a given host has four available NICs (e1000g0, 1, 2 and 3) and e1000g0 is the current primary NIC for the host, am I right to think that if I were going to aggregate at least two NICs together for increased bandwidth in a backup server role, I'd want to include e1000g0 as one of those NICs?

Posted by Ty Young on March 05, 2009 at 04:10 PM EST #

Hi Jay, I've been doing this for a while on various systems. What i have also sometimes done is assign multiple ip addresses to zones. This is simple, do an "add net ...... " twice. The only problem I've had so far with this is that outgoing connections from within the zone always come from one ip address, the first one that appears in the zone configuration file. For things such as active ftp connections, this throws a wrench in the system, with the client getting a connection request back from a different ip address. I haven't found a solution for this yet, so I'm open to suggestions, if anyone knows something.

Posted by Ian B on April 17, 2009 at 05:04 AM EDT #

Post a Comment:
  • HTML Syntax: NOT allowed

This blog copyright 2009 by Jay Danielsen