dladm is used for creating link aggregations on Solaris 10. This can be used on any nemo based driver. Unfortunatley on your T2000 unless you are running update 2 with the latest patch bundle you are stuck using ipge. The latest e1000g driver is included in KU 118833-20 or higher. Patch 123334 will take care of the transition for you.
nicky veitch's Weblog nicky veitch's Weblog
Tuesday Aug 29, 2006
Tuesday Aug 08, 2006
This is a walk through on how you might want to generate your own ssl certs and
and use those with a webserver with kssl on Solaris 10.
Start by setting up your own certificates
oaf207# cd /opt/SUNWwbsvr/CA
oaf207# openssl
oaf207# mkdir certs crl newcerts private
oaf207# echo "01" > serial
oaf207# cp /dev/null index.txt
oaf207# cp /etc/sfw/openssl/openssl.cnf .
oaf207# vi openssl.cnf
and change
dir = /etc/sfw/openssl # Where everything is kept
dir = /opt/SUNWwbsvr/CA
oaf207# openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 \
> -config openssl.cnf
Generating a 1024 bit RSA private key
....++++++
...++++++
writing new private key to 'private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
------
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:ie
State or Province Name (full name) [Some-State]:dublin
Locality Name (eg, city) []:clontarf
Organization Name (eg, company) [Unconfigured OpenSSL Installation]:sun
Organizational Unit Name (eg, section) []:perf
Common Name (eg, YOUR name) []:testuser
Email Address []:configure_ssl@sun.com
oaf207# openssl req -nodes -new -x509 -keyout newreq.pem -out newreq.pem -days 365 \
> -config openssl.cnf
Generating a 1024 bit RSA private key
....++++++
..................++++++
writing new private key to 'newreq.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [US]:ie
State or Province Name (full name) [Some-State]:dublin
Locality Name (eg, city) []:clontarf
Organization Name (eg, company) [Unconfigured OpenSSL Installation]:sun
Organizational Unit Name (eg, section) []:perf
Common Name (eg, YOUR name) []:testuser
Email Address []:configure_ssl@sun.com
oaf207# openssl x509 -x509toreq -in newreq.pem -signkey newreq.pem -out tmp.pem
Getting request Private Key
Generating certificate request
oaf207# openssl ca -config openssl.cnf -policy policy_anything -out newcert.pem \
> -infiles tmp.pem
Using configuration from openssl.cnf
6789:error:0E06D06C:configuration file routines:NCONF_get_string:no value:/on10/build-nd/G10U2B2/usr/src/common/openssl/crypto/conf/conf_lib.c:329:
group=CA_default name=unique_subject
Enter pass phrase for /opt/SUNWwbsvr/CA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Aug 4 15:08:50 2006 GMT
Not After : Aug 4 15:08:50 2007 GMT
Subject:
countryName = ie
stateOrProvinceName = dublin
localityName = clontarf
organizationName = sun
organizationalUnitName = perf
commonName = testuser
emailAddress = configure_ssl@sun.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
FC:8D:C6:7C:D5:92:13:45:0E:85:74:8F:E1:3C:C8:89:B2:29:89:17
X509v3 Authority Key Identifier:
keyid:6F:DF:38:7D:D1:E7:C6:B5:ED:8D:19:57:13:CC:C4:2F:C0:2E:64:C6
DirName:/C=ie/ST=dublin/L=clontarf/O=sun/OU=perf/CN=testuser/emailAddress=configure_ssl
@sun.com
serial:00
Certificate is to be certified until Aug 4 15:08:50 2007 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
oaf207# ksslcfg create -f pem -i `pwd`/newreq.pem -x 8080 -p \
> /opt/SUNWwbsvr/alias/password 443
oaf207# svcs |grep kssl
online 12:17:41 svc:/network/ssl/proxy:kssl-INADDR_ANY-443
oaf207#
Thursday Jun 01, 2006
Solaris 10 introduced the nemo framework for drivers and Solaris Nevada has more projects which build on said framework. In Update 2 of Solaris 10 support for data link aggregation was added which means we can build fat network pipes from most nics :) without any trunking software.
From the manual for dladm
The dladm command is used to configure data-links. A config-
ured data-link is represented in the system as a STREAMS
DLPI (v2) interface which may be plumbed under protocol
stacks such as TCP/IP. Each data-link relies on either a
single network device or an aggregation of devices to send
packets to or receive packets from a network.
Heres it on a galaxy but you can do the same on a t2000 using the new e1000g driver.
oaf316# ifconfig -a unplumb oaf316# dladm show-dev e1000g0 link: up speed: 1000 Mbps duplex: full e1000g1 link: up speed: 1000 Mbps duplex: full e1000g2 link: up speed: 1000 Mbps duplex: full e1000g3 link: up speed: 1000 Mbps duplex: fullNow we know what devices are available for our aggregation. We will make a an aggregation of 2 of the nics.
oaf316# dladm create-aggr -d e1000g0 -d e1000g3 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:c8:b0 (auto)
device address speed duplex link state
e1000g0 0:14:4f:1:c8:b0 1000 Mbps full up standby
e1000g3 0:14:4f:1:c8:b3 1000 Mbps full up standby
Agrregation completes and the link is in standby mode, next we need to plumb it.
oaf316# ifconfig aggr1 plumb oaf316# ifconfig aggr1 10.1.10.1 netmask 255.255.255.0 upregular ifconfig to setup the link. Lets check the device state now.
dladm show-aggr 1
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:c8:b0 (auto)
device address speed duplex link state
e1000g0 0:14:4f:1:c8:b0 1000 Mbps full up attached
e1000g3 0:14:4f:1:c8:b3 1000 Mbps full up attached
We add some more nics to the device while the device is up and running.
oaf316# dladm add-aggr -d e1000g1 -d e1000g2 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:c8:b0 (auto)
device address speed duplex link state
e1000g0 0:14:4f:1:c8:b0 1000 Mbps full up attached
e1000g3 0:14:4f:1:c8:b3 1000 Mbps full up attached
e1000g1 0:14:4f:1:c8:b1 1000 Mbps full up attached
e1000g2 0:14:4f:1:c8:b2 1000 Mbps full up attached
Now lets show off and remove a nic from the link
oaf316# dladm remove-aggr -d e1000g0 1
oaf316# dladm show-aggr 1
key: 1 (0x0001) policy: L4 address: 0:14:4f:1:c8:b3 (auto)
device address speed duplex link state
e1000g3 0:14:4f:1:c8:b3 1000 Mbps full up attached
e1000g1 0:14:4f:1:c8:b1 1000 Mbps full up attached
e1000g2 0:14:4f:1:c8:b2 1000 Mbps full up attached
heres the postinstall script which we use with some of our systems, you will have to change the IP.
# script to setup link aggregation on nics which are not in use
# it trys to ignore unsupported nics.
# This script can be used as part of a jumpstart.
#
# man dladm
PRE=/
[ -f /a/usr/sbin/dladm ] && PRE=/a
PATH=$PRE/usr/bin:$PRE/usr/sbin
export PATH
# ip for configured device to use eg 10.1.1.1
IP=10.1.1.1
# netmasks for configured device eg 255.255.255.0
Netmasks=255.255.255.0
# set this to "e1000g0 nge0 bge1" etc this can be left blank and we try to use
# other gld nics
NicsToUse=
showError() {
echo "$0: $1"
exit 1
}
# exit if no ip or netmask
[ -z "$IP" -o -z "$Netmasks" ] && showError "IP and Netmasks must be defined"
# check network devices exists
if [ -z "$NicsToUse" ]; then
ifconfig -a plumb 2>/dev/null
NicsToUse=`ifconfig -a |awk -F: '//dev/null
[ `dladm show-link $nic |grep -v -c legacy` ] && vNics="$vNics -d $nic"
done
# no nics supported by dladm
[ -z "$vNics" ] && showError "No supported nics on system"
# configure and plumb device
dladm create-aggr -R $PRE $vNics 1
[ $? != 0 ] && showError "error configuring aggr1 with dladm and $vNics"
ifconfig aggr1 plumb
ifconfig aggr1 $IP netmask $Netmasks up
[ $? != 0 ] && showError "error bringing up aggr1 with ifconfig $IP netmask $Netmasks "
# store nic details
echo $IP >$PRE/etc/hostname.aggr1
IP=`echo $IP|cut -f1-3 -d\.`.0
echo "$IP $Netmasks" >>$PRE/etc/netmasks
You will also want to increase the number of soft rings used by your aggregations. This can be done via /etc/system or via mdb as the default is 2 per interface.
oaf316# mdb -kw Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 ufs ip sctp usba fcp fctl nca random md lofs zfs nfs sppp crypto cpc fcip logindmux ptm ] > ip_soft_rings_cnt/W 8 ip_soft_rings_cnt: 0x2 = 0x8 > $qThis increases the number of soft rings from 2 to 8. If your aggregation is already plumbed you will need to replumb it to take advantage of the extra rings. To make this permanent you will need to add it to /etc/system
set ip:ip_soft_rings_cnt=8Be warned this will do it for each link after the next reboot. More to come on t2000 and link aggregation.
Monday Sep 19, 2005
We recently got some new hardware which has some nice logic in the bios which says "if I have a mbr and no one has pressed F12 to select network boot, then silently boot from the default boot device". This is not the behaviour I am used, normally if the default boot device is set to network and there are no offers then boot from the next device in the boot list. Unfortunatley the default boot device on these machines is hard coded to the disk!, this is really annoying when your trying to jumpstart the system remotely and have no physical access.
There are 2 options since Solaris with grub is installed on the system, we can wipe the boot record and force a boot from the default device or modify the grub menu.
Wiping the mbr - pro: easily done, cons if something goes wrong theres no way back
Modify Grub - pro: can revert back to default os, cons need hands on if things go wrong
This assumes you have a tftp server configured for jumpstart and are using dhcp, all you do is add the following to the menu on the system to be reinstalled in /boot/grub/menu.lst. Typically I place the entry below the timeout entry ie as the first entry and set the default boot entry to 0
# # default menu entry to boot default 0 # # menu timeout in second before default OS is booted # set to -1 to wait for user input timeout 10 # dhcp root (nd) kernel /I86PC.Solaris_11-15/multiboot kernel/unix - install dhcp -B console=keyboard,install_config=IP_INSTALL_SERVER:/PATH_TO_JUMPSTART_CONFIG/MACHINE_NAME,sysid_config=IP_INSTALL_SERVER:/PATH_TO_JUMPSTART_CONFIG/MACHINE_NAME,install_media=IP_INSTALL_SERVER:/PATH_TO_INSTALL_MEDIA/OS_BUILD,install_boot=IP_INSTALL_SERVER:/PATH_TO_INSTALL_MEDIA/OS_BUILD/boot module /I86PC.Solaris_11-15/x86.minirootThe entries for kernel and module can be found in the menu.lst.MAC-ADDRESS on your tftp server after running add_install_client. The above entry is used when you have a jumpstart profile, to use an interactive install (why?) the entries would be as follows
# # default menu entry to boot default 0 # # menu timeout in second before default OS is booted # set to -1 to wait for user input timeout 10 # dhcp root (nd) kernel /I86PC.Solaris_11-15/multiboot kernel/unix - install dhcp module /I86PC.Solaris_11-15/x86.miniroot
Simply what happens is, dhcp tell grub to configure the network device based on the dhcp settings. We set the root filesystem to be the network devices tftp directory. Load the kernel from the directory with the args of install and dhcp .
note
this will only work if you have a network device which is supported by the mini root.
Friday Sep 16, 2005
My laptop had network card which is almost supported by Solaris (read oem version of well known card), as of late I've been doing some work with jumpstart and my laptop has become my test machine. Since my card is unsupported I need to modify the miniroot so that I can pxe install the system. And I also need to add the entry as part of my postinstall script to allow me use the networking on my system.
The process of modifying the install miniroot is straight forward once you remember that the system is little endian so do the mods on a little endian box.
gunzip < solaris_build/boot/x86.miniroot >/tmp/miniroot lofiadm -a /tmp/miniroot /dev/lofi/2 # mount /dev/lofi/2 /mnt # echo 'iprb "pci8086,1050"\n' >>/mnt/etc/driver_aliases # tail -2 driver_aliases iprb "pci8086,1050" umount /mnt # lofiadm -d /dev/lofi/2 # gzip < /tmp/miniroot > solaris_build/boot/x86.miniroot
At some point over the last week I managed to swap the contents of my bootenv.rc for "hello world", well not quite, it actually contained bootfile='kernel/amd64/unix'. Theres nothing wrong with the entry per say, the problem is ALL the other entries had been over written. The system stayed up and continued running until I made the mistake of rebooting the box.
The symptoms of my over zealous redirect showed the following on the screen *snip* bios boot.... grub boot loader.. starts default boot of multiuser *snip* SunOS Release 5.11 Version XXX 64-bit Copyright 1983-2005 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. few lines of panic info and then back to bios, repeat as required. The solution was to boot a single user session, ie the failsafe session in gnome and edit the root partition. When booting the failsafe mini rooot asked if I wanted to mount the OS it found on the system. I agreed and my regular root filesystem was mounted as /a. I examined the contents of /a/boot/solaris/bootenv.rc and noticed the problem. I examined the contents of the miniroot bootenv.rc
*snip* setprop kbd-type US-English setprop ata-dma-enabled 1 setprop atapi-cd-dma-enabled 1 setprop ttyb-rts-dtr-off false setprop ttyb-ignore-cd true setprop ttya-rts-dtr-off false setprop ttya-ignore-cd true setprop ttyb-mode 9600,8,n,1,- setprop ttya-mode 9600,8,n,1,- setprop lba-access-ok 1 setprop prealloc-chunk-size 0x2000 setprop input-device 'keyboard' setprop output-device 'screen' */snip*ok no big problem, copy the file to /a/boot/solaris/bootenv.rc and just add the bootpath and bootfile. The bootpath can be got from the current device tree
ls -l /dev/dsk/c0d1s0 lrwxrwxrwx 1 root root 50 Sep 16 10:46 /dev/dsk/c0d1s0 -> ../../devices/pci@0,0/pci-ide@7,1/ide@0/cmdk@1,0:abased on this we add
bootpath='/pci@0,0/pci-ide@7,1/ide@0/cmdk@1,0:a'to the bootenvrc, since we want to boot 64bit add the entry which caused the problem in the 1st place
bootfile='kernel/amd64/unix'almost forgot to set the kbd type
setprop kbd-type 'UK-English'reboot and were back....
Tuesday Apr 19, 2005
I graduated college some time in the dark distant past and started working for Sun Ireland. The group I joined consisted of 2 engineers and an intern. This was the original Performance PIT, benchmarking each build of Solaris using perhaps a dozen benchmarks on a build by build basis. Roll on a number of years and the team has expanded to almost a dozen people with hundreds of benchmarks executed on each build of Solaris, to say nothing of the analysis. Welcome to the wonderful world of the Performance lifestyle.
In the last few months we have moved labs as we increase coverage and prepare to ramp up for testing of open solaris. The new lab (pictured above) provides ample parking for all our machines, although our site facilities may differ on the point.
Having seen the recent release of filebench to source forge by Richard I though I might post a bit about my experiences using the filebench framework. Filebench is a frame work which allows the user emulate an applications interaction with the filesystem aka personalities.
Varmail - multi threaded postmark style workload
File server - similair to Spec sfs , the nfs benchmark
Oltp - a database emulator modelled on Oracle 9i access patterns
Web server
Web proxy
Copy files
Create files
Random read
Random write
Single stream read
Single stream write
Multi stream write
By default filebench has the following predefined personalities
Application personalities
Micro-benchmarks
Filebench - getting started
I have only used Filebench on Solaris (x86/sparc) but I am eagerly waiting to get my hands on the Linux distro. To install on Solaris is the same as adding any other package,
# pkgadd -d . filebenchthis installs the package in /opt/filebench. To execute a single personality
# bin/filebench filebench> load webproxy 1641: 5.761: Usage: set $dir=1641: 5.761: set $filesize= defaults to 16384 1641: 5.761: set $nfiles= defaults to 1000 1641: 5.761: set $nthreads= defaults to 100 1641: 5.761: set $meaniosize= defaults to 16384 1641: 5.761: set $meandirwidth= defaults to 1000000 1641: 5.761: (sets mean dir width and dir depth is calculated as log (width, nfiles) 1641: 5.761: dirdepth therefore defaults to dir depth of 1 as in postmark 1641: 5.761: set $meandir lower to increase depth beyond 1 if desired) 1641: 5.761: 1641: 5.761: run runtime (e.g. run 60) filebench> set $dir=/tmp filebench> set $nfiles=100 filebench> create filesets 1641: 80.772: Fileset bigfileset: 100 files, avg dir = 1000000.0, avg depth = 0.3, mbytes=1 1641: 80.772: Creating fileset bigfileset... 1641: 80.853: Preallocated 86 of 100 of fileset bigfileset in 1 seconds filebench> create files 1641: 90.412: Creating/pre-allocating files filebench> create processes 1641: 97.513: Starting 1 proxycache instances 1644: 98.519: Starting 100 proxycache threads filebench> stats clear filebench> sleep 120 1641: 112.585: Running... 1641: 233.619: Run took 120 seconds... filebench> stats snap 1641: 271.376: IO Summary: 5986660 ops 36127.5 ops/s, (9507/1901 r/w) 103.7mb/s, 117us cpu/op, 0.1ms latency filebench> shutdown processes 1641: 309.257: Shutting down processes filebench> quit
