So you have a laptop running Solaris Nevada SXCE and want to "upgrade" to OpenSolaris before SXCE becomes
history and maybe run them side by side for a few builds, (uhmm) to get the transition sorted out.

There are several ways to achieve this, none of them are supported but this one is "relatively" clean to do.
Sun internal folks can download the build 127 usb iso from the usual sfbay location, and from here for the community.

http://genunix.org/distributions/indiana/osol-dev-127-x86.usb

What do we need:
You'll need two spare USB sticks, one of at least 1GB and the other of at least 4GB, you need to already be
running ZFS boot on your SXCE install. See http://blogs.sun.com/thinslice/entry/upgrade_to_all_zfs_root
for a great set of instructions from Iain Curtain to get to that point

Strategy:
We'll use usbcopy to put the OpenSolaris Build 127 liveusb iso onto the smaller stick, we'll boot that live
image and then install OpenSolaris on the larger stick. Then we put a new OpenSolaris BE on to the same rpool
used by your SXCE BEs and force them to play nice together for a while.

Let's go:
If you haven't made a backup of your rpool I suggest you do it now before we start,
I don't anticipate trouble but if it comes don't be unprepared.

Grab usbcopy if you don't already have it, best make it executable .

# /usr/sfw/bin/wget http://src.opensolaris.org/source/xref/livemedia/livemedia/usbcopy
# chmod a+rx usbcopy

Insert the smaller stick

# ./usbcopy osol-dev-127-x86.usb

Select the newly inserted stick.
Once complete, reboot the system from that stick, then also insert the second larger USB stick.

Install OpenSolaris on to it using the installer icon on the desktop
- DO **NOT** reboot yet.

Now the fun bits:

To avoid a collision of root pool names, if your zfs booted SXCE uses "rpool" for the name of your root
pool, you will need to rename the root pool on your newly installed 4Gb stick.

a/ export the pool
# zpool export rpool
b/ Obtain the numeric ID of the instance of "rpool" residing on the USB stick.
# zpool import

You will expect to see two pools named rpool, select the one you know to be the usb stick,
if in any doubt, use "eject -l" to tell you which USB sticks equate to which disk device
displayed by "zpool import".

# zpool import
pool: rpool
id: 5398210322707898534
state: ONLINE
config:

Rpool ONLINE
c6t0d0s0 ONLINE

c/ Import that ID under a new name, (this will make the stick unbootable without intervention but we can handle that).

# zpool import 5398210322707898534 Rpool

# zpool status
pool: Rpool
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
Rpool ONLINE 0 0 0
c6t0d0s0 ONLINE 0 0 0

Export the pool again.

# zpool export Rpool

Now halt the live usb instance and remove the small USB stick

Boot from the large USB stick, at the grub menu use the "e" key to edit the
grub menu and select the line "bootfs rpool/ROOT/opensolaris" then "e" to edit
that line, (please don't be tempted to also change the findroot line).
The "bootfs" line needs to use the new pool name, so it should read

"bootfs Rpool/ROOT/opensolaris"

Once done escape back to the grub menu and boot the modified entry
using the "b" key.

Now that we are booted in OpenSolaris, we will make a new OpenSolaris boot environment
on the real disk based rpool which already contains your SXCE BEs.

# beadm create -p rpool zOBE

You can of course name it "opensolaris" if you want but zOBE suits me.

Activate the new BE.

# beadm activate zOBE

Halt and remove the larger memory stick, but KEEP it as you may want it
later if you choose to do further updates of your SXCE environments
using "Live-Upgrade". You can do that, it will work fine however
you MUST remember that OpenSolaris is your BE master environment now,
to activate a new or upgraded SXCE BE do **NOT** use luactivate.
Instead boot manually via GRUB to test OR reboot OpenSolaris and use
"beadm activate" from the osol boot environment to activate the
new SXCE BE.

This is necessary as the SXCE grub appears to be poisonous to OpenSolaris
BEs and must not be propagated. If you do use luactivate you'll need the
"rescue" USB stick you have just put aside for later.

Boot from your newly activated OpenSolaris BE.

# beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
opensolaris - - 4.32G static 2009-11-14 08:58 (That's my 4Gb stick)
zABE - - 18.28G static 2009-11-07 23:42 (My old SXCE main BE)
zMBE - - 115.02M static 2009-10-22 09:06 (My old SXCE alternate BE)
zOBE NR / 3.21G static 2009-11-15 14:43 (The new OpenSolaris BE)

Create a new safety BE and then start grabbing bits as needed from your old environment.

You will definitely want to copy across /etc/hostid from your existing SXCE
or things get very messy at each reboot.

Mount your latest SXCE BE filesystem using the zfs commands (I'm not using beadm here,
but only as I haven't tested that yet, I expect it'll work fine but the usual caveats apply).

# zfs set mountpoint=/mnt rpool/ROOT/zABE
# zfs mount rpool/ROOT/zABE

# cp /mnt/etc/hostid /etc/hostid
# cp ... ..
etc.

Clean up:

# zfs umount /mnt
# zfs set mountpoint=/

Check that the OpenSolaris BE is sharing the swap and dump devs with the SXCE BEs.

# dumpadm -d /dev/zvol/dsk/rpool/dump
# dumpadm
Dump content: kernel pages
Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/georges-home
Savecore enabled: yes
Save compressed: on
#

# swap -l
No swap devices configured

Add it manually if needed

# swap -a /dev/zvol/dsk/rpool/swap

Edit /etc/vfstab to add the following line for swap.

/dev/zvol/dsk/rpool/swap - - swap - no -

Now you are good to go with a reboot, then manually select an SXCE BE at
the GRUB menu, using lustatus you should see ONLY the SXCE BEs, just try
to remember that this environment is on it's way to being deprecated so
don't allow GRUB or GRUB menus to be propagated from here. SO, don't use
luactivate or installgrub at all when booted SXCE.

So back to your OpenSolaris BE, it's time to use the pkg_manager to go
fetch the packages you want, from the dev repository. (We are on the bleeding
edge here aren't we, since we're likely on snv_127 in both SXCE and Osol .

Enjoy.

GNOME was up-revved from 2.26 (snv_123) to 2.28 (snv_124).

If like me you love the new login sound which came with Nevada build 124, here's how to change it.

Here is the config file which specifies the startup sound.

/usr/share/gnome/autostart/libcanberra-login-sound.desktop
----------------------------
[Desktop Entry]
Type=Application
Name=GNOME Login Sound
Comment=Plays a sound whenever you log in
Exec=/usr/bin/amd64/canberra-gtk-play --id="desktop-login" --description="GNOME Login"
OnlyShowIn=GNOME;
AutostartCondition=GNOME /desktop/gnome/sound/event_sounds
X-GNOME-Autostart-Phase=Application
X-GNOME-Provides=login-sound
----------------------------

Mount your old boot environment and copy over the snv_123 or earlier "desktop-login" sound file.

I'm using the SXCE Solaris distro, OpenSolaris paths will differ.

cp /.alt.zABE//usr/share/sounds/freedesktop/stereo/desktop-login.ogg /usr/share/sounds/freedesktop/stereo/desktop-login-trad.ogg
Run the command to test this and then modify the config file to match.

/usr/bin/amd64/canberra-gtk-play --id="desktop-login-trad"

Courtesy of a guest list place won via Suzanne Vega.com, my wife and I enjoyed Suzanne's live London gig last night.
The show was a part of the itunes live festival.

http://www.ituneslive.co.uk/suzannevega/

I've been a great fan of Suzanne's erudite lyrics and quirky vocal style since her first Album, Her latest Beauty and Crime is yet another classic example.

Way to go Suzanne !!

Since OpenSolaris is releasing today, I thought I'd take a few lines to recount how we can use DTrace to snoop in Streams data. Here DTrace helped me to quickly and cleanly understand a specific Streams bug (6230528).

My usual business is Sustaining Solaris Streams and TCP/IP code, but I occasionally moonlight supporting legacy communications protocols still used by the Telco world. In that capacity I was involved in certifying the legacy Solstice X.25 9.2 product as supportable on the (then) upcoming Solaris 10 release.

The X.25 product is highly configurable, and one of it's more esoteric configs encapsulates the IP protocol over an X.25 link. To do this a userland daemon, (x25netd) opens a stream to the product's ixe driver, (a pseudo NIC driver) and uses the Streams I_LINK ioctl to create a stream to the X.25 driver below. It then uses ifconfig to bring up the pseudo NIC instance, pushing the IP module onto the stream, and voila IP over X.25.

Well IP over X.25 functioned just fine, but occasionally when you brought the X.25 product down, the system paniced in Streams code, doing putnext() of a Streams message downstream from the ixe driver to the X.25 layer.

The clear clue was this console message at every close: 
WARNING:KERNEL: munlink: could not perform unlink ioctl, closing anyway (22)

The I_UNLINK ioctl sent downstream by x25netd when it was being shutdown was failing with EINVAL.

I needed to determine what syndrome caused the stream below ixe to evaporate leading to the panic, while the I_UNLINK ioctl intended to safely demolish that stream had failed (?)

DTrace is our friend here, we can use it to slice into the Stream and select what we want to see, without adding debug and recompiling multiple Streams modules and drivers, in this case from more than one code base.  DTrace has to be the world's most flexible debugging tool, Victorinox watch out.

Let's write a D script restricting the stuff we view to the x25netd process, and look at the munlink() and ioctl activity.
Examining the code we can see the functions involved in performing a Streams ioctl, we instrument those, looking for return codes and Streams errors.
Additionally we instrument the general purpose Streams routine for moving data putnext(), but we restrict what we see to only M_IOCTL Streams messages and potential replies to M_IOCTL.

#!/usr/sbin/dtrace -s


inline int M_IOCTL = 0x0e;            /* ioctl; set/get params */
inline int M_IOCACK = 0x81;            /* acknowledge ioctl */
inline int M_IOCNAK = 0x82;            /* negative ioctl acknowledge */

fbt::strdoioctl:entry
/ execname == "x25netd" /               /* All Streams ioctls sent by x25netd */
{
        /* Separator */
        printf("\n");
}

fbt::strdoioctl:entry
/ execname == "x25netd" /               /* All Streams ioctls sent by x25netd */
{
        icp = (struct strioctl *) arg1;
        printf("IOCTL %d (0x%x), length %d", icp->ic_cmd, icp->ic_cmd, icp->ic_len);
}

fbt::strdoioctl:return
/ execname == "x25netd" /               /* Return code in arg1 */
{
        printf("RETURN=%d errno=%d pid=%d", arg1, errno, pid);
}

fbt::munlink:return
/ execname == "x25netd" /               /* munlink() returns?? */
{
        printf("RETURN=%d errno=%d pid=%d", arg1, errno, pid);
}

fbt::putiocd:return
/ execname == "x25netd" /
{
        printf("RETURN=%d errno=%d pid=%d", arg1, errno, pid);
}

fbt::putnext:entry
/ execname == "x25netd" && (((mblk_t *) arg1)->b_datap->db_type == M_IOCTL)/
{
        iocp = (struct iocblk *) ((mblk_t *) arg1)->b_rptr;
        printf(" putnext msg from: %s:: M_IOCTL ioctl %d (0x%x) count %d",
            (stringof((queue_t *) arg0)->q_qinfo->qi_minfo->mi_idname),
            iocp->ioc_cmd, iocp->ioc_cmd, iocp->ioc_count);
}

fbt::putnext:entry
/ execname == "x25netd" && (((mblk_t *) arg1)->b_datap->db_type == M_IOCACK)/
{
        iocp = (struct iocblk *) ((mblk_t *) arg1)->b_rptr;
        printf(" putnext msg from: %s:: M_IOCACK ioctl  %d (0x%x) count %d",
            (stringof((queue_t *) arg0)->q_qinfo->qi_minfo->mi_idname), 
            iocp->ioc_cmd, iocp->ioc_cmd, iocp->ioc_count);
}

fbt::putnext:entry
/ execname == "x25netd" && (((mblk_t *) arg1)->b_datap->db_type == M_IOCNAK)/
{
        iocp = (struct iocblk *) ((mblk_t *) arg1)->b_rptr;
        printf(" putnext msg from: %s:: M_IOCNAK ioctl %d (0x%x) count %d",
            (stringof((queue_t *) arg0)->q_qinfo->qi_minfo->mi_idname), 
            iocp->ioc_cmd, iocp->ioc_cmd, iocp->ioc_count);
}

fbt::strgeterr:return
/ execname == "x25netd" /
{
        printf("RETURN=%d errno=%d pid=%d", arg1, errno, pid);
}

fbt:ixe:ixewput:entry                   /* What M_IOCTLs arrive in ixe */
/ execname == "x25netd" && ((mblk_t *) arg1)->b_datap->db_type == M_IOCTL /
{
        iocp = (struct iocblk *) ((mblk_t *) arg1)->b_rptr;
        printf(" ixe M_IOCTL %d (0x%x) received, count %d", iocp->ioc_cmd,
            iocp->ioc_cmd, iocp->ioc_count);
}

Having started the X.25 product, the associated IP over X.25 NIC was tested, then I started the D script and waited for all the DTrace probes to be established. It was time to gather the data, by stopping the IP over X.25 device.

  0  13220      strdoioctl:entry 

  0  13220      strdoioctl:entry IOCTL 21261 (0x530d), length 24
  0  11497      putiocd:return RETURN=0 errno=0 pid=1763
  0   3511      putnext:entry  putnext msg from: strwhead:: M_IOCTL ioctl 21261 (0x530d) count 24
  0   3511      putnext:entry  putnext msg from: ip:: M_IOCNAK ioctl 21261 (0x530d) count 0
  0  13221      strdoioctl:return RETURN=22 errno=0 pid=1763
  0   9815      munlink:return RETURN=0 errno=0 pid=1763

From this it was apparent that the munlink was actually SUCCEEDING, but that the I_UNLINK ioctl was being NAK'ed by the IP module and never reaching the ixe driver.
The end result was that the STREAMS framework part of the unlink processing had occurred, and the linkage below ixe was gone, but nothing had told ixe that the stream below was no more. If a message from IP was in flight as the pseudo NIC was ifconfig'ed down, a system panic was likely. The munlink code has a special case to handle failure of the I_UNLINK ioctl upon closure of the stream.

         error = strdoioctl(stp, &strioc, NULL, FNATIVE, 
             K_TO_K | STR_NOERROR | STR_NOSIG, crp, rvalp); 
  
         /* 
          * If there was an error and this is not called via strclose, 
          * return to the user. Otherwise, pretend there was no error 
          * and close the link. 
          */ 
         if (error) { 
                 if (flag & LINKCLOSE) { 
                         cmn_err(CE_WARN, "KERNEL: munlink: could not perform " 
                             "unlink ioctl, closing anyway (%d)\n", error); 
                 } else { 
                         link_rempassthru(passq); 
                         mutex_enter(&stpdown->sd_lock); 
                         stpdown->sd_flag &= ~STRPLUMB; 
                         cv_broadcast(&stpdown->sd_monitor); 
                         mutex_exit(&stpdown->sd_lock); 
                         mutex_exit(&muxifier); 
                         return (error); 
                 } 
         } 

This allowed code later in munlink() to destroy the Streams linkage despite the failure of the ioctl. The special case doesn't seem like a good idea, however the root cause of this problem is elsewhere.

The IP module has no exclusive rights to the I_UNLINK ioctl, therefore it should be passing it downstream rather than NAK'ing it, and this is the information which the D script made clear immediately.
In fact it turned out that when the FireEngine code, (TCP/IP merge and rewrite to use vertical perimeters) was putback in S10, it altered IP's ioctl handling, and where IP consumes an ioctl when used as a Streams driver, it was assuming the right to do so, when pushed as a Streams module, on another driver's stream!
The fix was to mark the subset of the ioctls consumed by IP as a driver which are generic Streams ioctls, with a flag to indicate they must be passed downstream when IP is pushed as a Streams module.
Search the source of OpenSolaris for IPI_PASS_DOWN.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Technorati Tag: DTrace

A visit to Prague last week to run a Sparc dump analysis class reminded me that I've forgotten more than half of what I knew about Sparc CDA. There is a very real danger in specialisation, should you neglect to deliberately step outside your role regularly to explore or simply re-imprint once over learned skills.

Currently working in Solaris sustaining, (the OP/N1 RPE team), I play in the STREAMS framework, Streams modules such as TCP, IP and the TL driver, and I'm leading the support of several legacy products, (X.25, OSI, FTAM, CMIP, Frame Relay, TMN-Suite et-al).

An Australian born and raised European compu-nomad, I've found my way to the U.K. this last year, following a five year stint in France.

A hardware engineer by training, with a background servicing digital radar for the Australian Airforce, I did some mainframe CE and data communications work, as well as chip and transistor changing, before joining Sun in December 1987.

Yup count 'em, 16 years.

Sun is the only kind of envoronment I can survive for that long, I'm not generally one to stay in any place longer that 5 years, but in Sun I've swapped roles every 4 years or so, and pretty much picked and chosen what I want to do and when. (Recessions and burst bubbles aside the choice of time and place has been mine).

So having done hardware, training, people management and tech management in Australia I chose to go to France in a software sustaining role on Streams based data comminications products.

Two main factors drove that decision, providing an education money can't buy for my then 2 year old daughter, and filling that little hole in my CV around C code. While in France I ran several trainings in Sparc crash dump analysis and did some little dabbling back in technology management.

That much aside, welcome to my blog. I plan to update on what I'm working on occasionally , along with a little home spun philosophy.