Thursday August 06, 2009

Sometimes in the course of being a system administrator it is useful to know what file systems are being mounted and when and what mounts fail and why. While you can turn on automounter verbose mode that only answers the question for the automounter.

Dtrace makes answering the general question a snip:

: exdev.eu FSS 24 $; cat mount_monitor.d                         
#!/usr/sbin/dtrace -qs

fbt::domount:entry
/ args[1]->dir /
{
        self->dir = args[1]->flags & 0x8 ? args[1]->dir : 
              copyinstr((intptr_t)args[1]->dir);
}
fbt::domount:return
/ self->dir != 0 /
{
        
        printf("%Y domount ppid %d, %s %s pid %d -> %s", walltimestamp, 
              ppid, execname, self->dir, pid, arg1 == 0 ? "OK" : "failed");
}
fbt::domount:return
/ self->dir != 0 && arg1 == 0/
{
        printf("\n");
        self->dir = 0;
}
fbt::domount:return
/ self->dir != 0 && arg1 != 0/
{
        printf("errno %d\n", arg1);
        self->dir = 0;
}
: exdev.eu FSS 25 $; pfexec /usr/sbin/dtrace -qs  mount_monitor.d
2009 Aug  6 12:57:57 domount ppid 0, sched /share/consoles pid 0 -> OK
2009 Aug  6 12:57:59 domount ppid 0, sched /share/chroot pid 0 -> OK
2009 Aug  6 12:58:00 domount ppid 0, sched /share/newsrc pid 0 -> OK
2009 Aug  6 12:58:00 domount ppid 0, sched /share/build2 pid 0 -> OK
2009 Aug  6 12:58:00 domount ppid 0, sched /share/chris_at_play pid 0 -> OK
2009 Aug  6 12:58:00 domount ppid 0, sched /share/ws_eng pid 0 -> OK
2009 Aug  6 12:58:00 domount ppid 0, sched /share/ws pid 0 -> OK
2009 Aug  6 12:58:03 domount ppid 0, sched /home/tx pid 0 -> OK
2009 Aug  6 12:58:04 domount ppid 0, sched /home/fl pid 0 -> OK
2009 Aug  6 12:58:05 domount ppid 0, sched /home/socal pid 0 -> OK
2009 Aug  6 12:58:07 domount ppid 0, sched /home/bur pid 0 -> OK
2009 Aug  6 12:58:23 domount ppid 0, sched /net/e2big.uk/export/install/docs pid 0 -> OK
2009 Aug  6 12:58:23 domount ppid 0, sched /net/e2big.uk/export/install/browser pid 0 -> OK
2009 Aug  6 12:58:23 domount ppid 0, sched /net/e2big.uk/export/install/cdroms pid 0 -> OK
2009 Aug  6 12:59:45 domount ppid 8929, Xnewt /tmp/.X11-pipe/X6 pid 8935 -> OK

In particular that last line if repeated often can give you a clue to things not being right.

Sunday May 10, 2009

I have made a change to up access hours script for my Sun Rays. Now the access file can also contain a comma separated list of Sun Ray DTUs so that the control is only applied to those DTUs:

: pearson FSS 3 $; cat /etc/opt/local/access_hours 
user1:2000:2300:P8.00144f7dc383
user2:2000:2300:P8.00144f57a46f
user3:0630:2300
user4:0630:2300
: pearson FSS 4 $; 

The practical reason for this is that it allows control of DTUs that are in bedrooms but if the computer is really needed another DTU can be used for homework.

Now that bug 6791062 is fixed the script is safe to use in nevada.

The script is where it always was.

Sunday February 15, 2009

I've managed to upgrade my home server to build 108 which is an important mile-stone for me as it has the fix for:

6763600: nautilus becomes unusable on a system with 39000 snapshots.

Which was rendering nautilus close to unusable for any users who moved out of their lofs automounted home directories. In partilcular any attempt to use it to manage the photo directories was painful.

However all was not smooth as again I hit this bug:

6795892: Sun Ray X servers (both Xsun and Xorg) suffer network problems in snv_106

but since I was expecting this I tried the workaround from bug 6799655 which is the same as the one for 6763600:

In /etc/sock2path change the following lines:

    2   2   0   tcp
    2   2   6   tcp

    26  2   0   tcp
    26  2   6   tcp

    2   1   0   udp
    2   1   17  udp

    26  1   0   udp
    26  1   17  udp

to:

    2   2   0   /dev/tcp
    2   2   6   /dev/tcp

    26  2   0   /dev/tcp6
    26  2   6   /dev/tcp6

    2   1   0   /dev/udp
    2   1   17  /dev/udp

    26   1  0   /dev/udp6
    26   1  17  /dev/udp6

While this got the Sun Rays up it also stopped named from working, spewing errors like this:

Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] 71/Protocol error
Feb 15 15:10:39 pearson named[15558]: [ID 873579 daemon.error] socket.c:4315: unexpected error:

So have had to revert to some internal to Sun binaries that work around this while the underlying bug is fixed. It is slighly worring as I'm left wondering what other victims are out there. One I have already found is ntp, which is a known bug:

6796948: NTP completely broken by I_SETSIG semantics change in snv_106

I suspect that the system will have to revert to build 105 soon.

Wednesday February 04, 2009

I managed to buy 2 Sun Ray 2's off Ebay and one of them was is now in place in the living room driving our 40” TV.

Combine this with a KeySonic wireless mini keyboard and the DTU does not only act as a photo frame. The Sun Ray unit is attached to the underside of the shelf as the top unit in the pile is a Virgin cable TV recorder which does not like having anything on top blocking the air flow. Thanks to the Sun Ray 2 being so light 5 strips of sticky back velcro do the trick so well that it really is going nowhere to the point that I could not remove it to plug the USB keyboard adapter directly in the back of the unit. The keyboard adapter has a button you have to press once plugged in to pair it with the keyboard. Alas with the Sun Ray in this configuration the button faces upwards. So there is a short USB cable hidden back there.

Networking is provided via Ethenet over mains.

The keyboard has impressive range and a really nice touch pad that pretends to have a scroll wheel down one side. However I've not yet got the keyboard map for it right but it only arrived an hour ago so there is time.

Wednesday January 07, 2009

Having installed a Sun Ray in my daughters bedroom I am now faced with the inevitable problem of her being online all night not getting any sleep and then being generally grumpy. The irony here is that I was sent an email asking how I handle access control to the DTUs and I said I just trusted the children to be sensible (what was I thinking!).

So a solution was required that gave access to the systems only between certain hours. The hours would depend on the user and would have to not loose all their “work” in case this was a late night finishing their homework session.

After asking around no one came back to me and said how it can be done so I wrote my own script. It works by having a file that contains lines with a format

user:starttime:endtime

The times are specified in 24 hour format and only accurate to the minute.

# cat /etc/opt/local/access_hours             
user1:1915:1900
user2:0630:2300
user3:0630:2230
user4:0630:2000
# 

The top line is just really for testing only not allowing access from 1900 to 1915. Then you need a user who has system admin privs which does not have a crontab file. Since I already have a kroot role I'm overloading this. Running the script as with the -c flag and the name of the user will write the crontab file. Note it also writes an entry to keep the crontab file uptodate on an hourly basis.

# /usr/local/sbin/check_access_hours -c kroot
# crontab -l kroot
46 * * * * /usr/local/sbin/check_access_hours -c kroot
00 19 * * * /usr/local/sbin/check_access_hours user1
00 23 * * * /usr/local/sbin/check_access_hours user2
30 22 * * * /usr/local/sbin/check_access_hours user3
00 20 * * * /usr/local/sbin/check_access_hours user4
#  

Finally I added a line to the utaction script that is already run for every user when they connect to a Sun Ray DTU:

if ! /usr/local/sbin/check_access_hours -t 0 $1
then
        exit 1
fi

The way it disallows access is that it adds the DTU's IP address to the ipfilter, which you have to have configured, so that all traffic from the DTU is blocked. It also submits an at(1) job to run 2 minutes in the future to remove the block so that the Sun Ray can burst back into life. The effect is that the user can no longer use any Sun Ray outside of the defined hours. But after about 2 minutes the DTU is usable again by others or indeed as a photo frame.

A word of warning. Having got all this running the system has paniced twice which is disappointing on one level, that it panics, but pleasing on another, I've found a bug that can now be fixed. The bug is:

6791062: System panic in ip_tcp_input when a rule is added to ipfilter

I look forward to the fix!

The script is here but check that that bug has been fixed before you use it.

Tuesday December 16, 2008

Here is some shell code I had forgotten about:

if (( $(date +%m) == 12 ))
then
        ( while  (( $(date +%d) >= 15 ))
        do
        test -f /var/run/snow-opts && snow_opts=$(</var/run/snow-opts)
        /opt/csw/bin/xsnow ${snow_opts:--santa 2}
        done ) &
fi

all for the the Sun Ray picture frames. The kids, all of us, love it.

Tuesday June 24, 2008

Our Sun Ray upgrade strategy hiccuped and now has both the SPARC systems running the same release, which when it is nv92 is nothing to complain about:

: enoexec.eu FSS 1 $; uname -a

SunOS enoexec 5.11 snv_92 sun4v sparc SUNW,SPARC-Enterprise-T5220

: enoexec.eu FSS 2 $;

The reason for the hiccup was two fold. Once we had established that the T5220 was a perfect Sun Ray server and managed to find and file some really nasty bugs found because we were using it a far sighted director agreed to fund one long term. This left the old Sun Fire system looking like a very large bit of tin, burning lots of power and taking up a lot of space while only providing one Sun Ray server. So that has now been replaced with a V890:

: estale.eu FSS 1 $; uname -a
SunOS estale 5.11 snv_92 sun4u sparc SUNW,Sun-Fire-V890
: estale.eu FSS 2 $; 

Since this was fresh hardware it was freshly installed and was to be the build 92 server while the T5220 served build 91 and would at some point serve build 94. That was until we diagnosed that we were hitting a bug on the T5220 which made it stall sometime for minutes that is fixed in build 92 so we have both systems running build 92.

Saturday June 21, 2008

At last the rest of the bits of “gaim” that disappeared from Solaris when it moved to be “pidgin” have returned in Nevada build 92. I'm talking about “purple-remote” which is the program that replaces “gaim-remote” and thus allows me once again to set my away message using “utaction” so when I disconnect from my Sun Ray session my IM status is automatically set as well.

If you take the script that I wrote last time and do a global edit changing “gaim-remote” to be “purple-remote” it will work. Something I realise now but did not then was that you only need one ut-action command to handle both connection and disconnection so this will do it:

utaction -d "purple-remote 'setstatus?status=away&message=Away from Sun Ray'" -c "${HOME}/bin/sh/ut-where"

Friday April 04, 2008

One of the great benefits of running Sun Rays at home is having the sessions always there. Just plug in the card and you get your session as if you were never away. However that also allows you to leave an application chewing CPU cycles when you are away. So to keep the interactive experience as good as possible I employ the same techniques described in “Using Solaris Resource Manager With Sun Ray” blueprint. For a long while I've wondered why IT don't do this. The keepers of our Sun Ray do and it works a treat. Which is a good thing when you share a Sun Ray Server with Tim.

Instead of setting the number of shares up to a specific value I use a multiplier so that those active on a Sun Ray get 10 times the number of shares that they would by default. While this works well it still leaves a significant load on the system from certain applications, specifically flash animations that are left running endlessly playing the games that were being played when the users card was removed. The fair share scheduler does it's thing to make CPU allocation fair but the memory use of those otherwise idle firefox sessions is significant.

So I've taken a leaf out of the BOFH and apply some special sanctions to those processes. Alas I may not get a job with the BOFH as my sanctions are simply to pstop(1) the copies of firefox associated with the user and DISPLAY when they detach and then prun(1) them when the user reconnects. I wondered about using memory resource caps to limit the memory but that would leave the systems rcapd(1M) battling the memory usage of the firefox processes which are not displaying anything anyway. In the unlikely event that any of the users are using their firefox sessions to simulate nuclear fission or crack SSL so would rather they kept running I'm sure they will get back to me.

So the script I have for doing this is slightly more complex than the one from the Blueprint. Since it has to err on the side of caution when stopping users firefox sessions. To do that it uses pargs(1) to make sure that the firefox sessions are really for this display. In practice I am the only person who might remote display a firefox session from here and even that is unlikely but it is the principle. The impact on the system of not trying to run all the disconnected firefox sessions is amazing.

Monday March 31, 2008

The downside is that one day you will ride to work only to find you have an empty pass holder in you pocket.

The upside is that you can check and see that you have left your pass in your Sun Ray @ home:

: enoexec.eu FSS 1 $;  utwho -c | grep 'JavaBadge.*cg13442'

 71.0 JavaBadgeNP.4090009c2311b2071914 cg13442  129.150.116.130 P8.00144f7dc334

: enoexec.eu FSS 2 $; 

So I know it is not lost. Now the problem is that it contains all the cash for lunch in a “cashless office”.

Tuesday March 25, 2008

One of my users today had a bit of a hissy fit today when she plugged in her USB thumb drive into the Sun Ray and it did nothing. That is it did nothing visible. Behind the scenes the drive had been mounted somewhere but there was no realistic way she could know this.

So I need a way to get the file browser to open when the drive is inserted. A quick google finds " "USB Drive" daemon for Sun Ray sessions" which looks like the answer. The problem I have with this is that it polls to see if there is something mounted. Given my users never log out this would mean this running on average every second. Also the 5 second delay just does not take into account the attention span of a teenager.

There has to be a better way.

My solution is to use dtrace to see when the file system has been mounted and then run nautilus with that directory.

The great thing about Solaris 10 and later is that I can give the script just the privilege that allows it to run dtrace without handing out access to the world. Then of course you can then give that privilege away.

So I came up with this script. Save it. Mine is in /usr/local which in turn is a symbolic link to /tank/fs/local. Then add an entry to /etc/security/exec_attr, subsisting the correct absolute (ie one with no symbolic links in it) path in the line.

Basic Solaris User:solaris:cmd:::/tank/fs/local/bin/utmountd:privs=dtrace_kernel

This gives the script just enough privileges to allow it to work. It then drops the extra privilege so that when it runs nautilus it has no extra privileges.

Then you just have to arrange for users to run the script when they login using:

pfexec /usr/local/bin/utmountd

I have done this by creating a file called /etc/dt/config/Xsession.d/utmountd that contains these lines:

pfexec /usr/local/bin/utmountd &
trap "kill $!" EXIT

I leave making this work for uses of CDE as an exercise for the reader.

Friday November 23, 2007

With the addition of the photo frame and a second Sun Ray the home server was beginning to struggle. The ideal (realistic) solution would be a multi core CPU but the lack of power management on those, at least the ones that will fit in the existing system, puts me off. However since most of the performance issues have been due to lack of memory there was a way out.

If you recall the motherboard in the system is ASUS M2NPV-VM which has four memory slots, however the system was supplied with a Zalman 8000 Low Profile Cooler which interferes with one memory slot reducing the number of available slots to 2 (the slots have to be used in pairs). The reason given for the optional cooler is to make the system quieter. So the only ways to increase the memory would be to replace the existing DIMMS with larger 2G DIMMS or swap the CPU fan for the default one, which I still have, and then add the extra memory. The alternative would be to unteach the family that it is OK just to pull your card out of the system when you walk away, now what is the fun of that?

It seems like a no brainer to me. I can always put the system in another room so the noise does not get me down.

So I have now added 4G of RAM, giving 6G total and put the AMD cooler on the system, so far so good:

Last login: Fri Nov 23 17:37:41 2007 from gmp-ea-fw-1.sun
Sun Microsystems Inc.   SunOS 5.11      snv_78  October 2007
: pearson FSS 1 $; prtconf | head -3
System Configuration:  Sun Microsystems  i86pc
Memory size: 6111 Megabytes
System Peripherals (Software Nodes):
: pearson FSS 2 $; batstat -t
Thermal zone: \_TZ_.THRM, temperature = 40C, critical = 75C
        Active Cooling: 73
: pearson FSS 3 $; 

Once again battling with the hardware really brought home just how fantastic the designs of Sun hardware is. This could have been designed by Citroen¹ given how difficult it is to replace bits. Even if I had not had to remove the CPU fan to add the memory I still would have had to remove all the disks, the main power supply the fan power cable just to add a DIMM.

What is surprising is that the system is very much quieter now than is used to be and even with the original AMD fan.

Tuesday June 12, 2007

Here is a change. The lab folks have added a third system to the group of Sun Ray servers we use and it is an AMD based system. So now the rolling upgrades will be every six weeks and I get to spend four weeks on SPARC and two weeks on AMD. Should flush out any problems with moving a home directory between architectures pretty quickly:

: eagain.eu FSS 1 $; uname -a
SunOS eagain 5.11 snv_66 i86pc i386 i86pc
: eagain.eu FSS 2 $; 

Friday June 08, 2007

I've been a bit remiss with the Good Morning Build updates over the last few weeks. Not because the updates are not happening but because there are only so many hours in the day.

: estale.eu FSS 1 $; uname -a
SunOS estale 5.11 snv_65 sun4u sparc SUNW,Sun-Fire
: estale.eu FSS 2 $; 

Build 65 has been on the server for a while now, so long that build 66 is upon us. However I thought I would bother to post this as I have just noticed that the volume control on the desktop now works on Sun Ray. A small step but with every build the Solaris Desktop gets better and better.

Tuesday February 06, 2007

New on a Sun Ray server near me, well would be if I was not at home but that is being picky. On the Sun Ray server I am using:

: estale.eu FSS 1 $; uname -a
SunOS estale 5.11 snv_57 sun4u sparc SUNW,Sun-Fire Solaris
: estale.eu FSS 2 $; 

Build 57. It did not have the svc:/system/hal:default running which in turn causes gnome-vfs-daemon to dump core.

The admins must have got a new module to solve Bug ID: 6519415 panic in apush_iocdata() from the development gate. Indeed they did:

: estale.eu FSS 4 $; what  /kernel/drv/sparcv9/sad 
/kernel/drv/sparcv9/sad:
        SunOS 5.11 onnv-gate:2007-02-04 October 2007
        SunOS Internal Development:  gk 2007-02-04 [onnv-gate]
: estale.eu FSS 5 $;

So far everything looks good. The updates to thunderbird are interesting.

Monday November 27, 2006

I finally got around to installing the Sun Ray software on the new Home Server. This was less simple than I had hoped. For some reason the configuration script would not add the correct settings to the dhcp server. However following the instructions over on the Think Thin blog for adding the settings manually. Once I did this the Sun Ray would still not work, getting a 26 D error, which if I understood it right meant that the appliance was not connecting to the Xserver.

Guess who had disabled the X server on the system? Well I did not need an X server on a headless system so I had disabled it. Thankfully restarting it is a snip:

# svcadm enable svc:/application/graphical-login/cde-login:default

and suddenly the Sun Ray burst into life, running build 53 it flies along. Now I'm off to Ebay to see if I can pick up some appliances, no bidding them up now.

Tags: Sun Ray home server solaris

Tuesday October 17, 2006

My previous use of gaim-remote and utaction to update my IM status automatically was good but I kept not getting invited to lunch when I came into the office as I forgot to reset the message to be “In the Office” or slightly less irritatingly would get invited to lunch when I was a home.

Since for the common case I only really work in two locations, the office or at home and the Sun Ray at home does not change often if I can figure out which Sun Ray I am currently connected to I could set the message automatically.

First you need to run this command to get the identity of the Sun Ray that is at home (or in any other location that is “special”):

utwho -c | nawk -v d=${DISPLAY#:} '$1 == d { print $5 }'

Now write that into a file in your home directory followed by a colon and then the description you wish to have appear when you are connected via that Sun Ray:

utwho -c | nawk -v d=${DISPLAY#:} '$1 == d {OFS=":"; print $5, "Working @ Home" }' >> ~/.ut_locations

You can populate that file with any number of specific locations.

Now put this script somewhere on your path called ut-where

#!/bin/ksh -xp

function get_location
{
        nawk -F: '$1 == "'$1'" { print $2 }' ~/.ut_locations
}


THIS_APPLIANCE=$(utwho -c | nawk -v d=${DISPLAY#:} '$1 == d { print $5 }')

THIS_LOC=$(get_location ${THIS_APPLIANCE})

gaim-remote 'setstatus?status=available&message='"${THIS_LOC:-At Office}"

As you can see it will set the default message to be “At Office” if it can't find the named Sun Ray otherwise it used the second colon separated field as the message to use for gaim.

Then update the ut-gaim script to be:

#!/bin/ksh -p
utaction -d "gaim-remote 'setstatus?status=away&message=Away from Sun Ray'" > /dev/null 2>&1 &
utaction -c "/home/cg13442/bin/sh/ut-where" > /dev/null 2>&1 &
wait

All the above assumes that you have /opt/SUNWut/bin in your path.

Now when each Sun Ray appliance has a build in GPS that you can query to find it's location this won't be so hard to do in the more general case.^*

Tags: Instant Messageing Sun Ray gaim

Wednesday August 23, 2006

Now that gaim-remote is in build 46 I can shamelessly steal an idea that Darren mentioned to me one lunch time.

Using this short script:

#!/bin/ksh -p
utaction -d "gaim-remote 'setstatus?status=away&message=Away from Sun Ray'" > /dev/null 2>&1 &
utaction -c "gaim-remote 'setstatus?status=available'" > /dev/null 2>&1 &
wait

I have my status in all the IM sessions that I have automatically set when I remove my card from my Sun Ray and reset when I return.

This assumes you have /opt/SUNWut/bin in your path.

Short and sweet.

Tags: solaris instant messaging Sun Ray

Thursday November 17, 2005

I've been on a course so have been away from my normal desk and Sun Ray server. Since our Sun Ray server runs nevada and is not part of the main IT setup when I plug my card in I have to login to the IT system and then use the utswitch command to switch to my server.

Then each time I use a new Sun Ray I have to the utwitch again.

However this is a better way.

ON logging into the IT system I do:

utaction -c "/opt/SUNWut/bin/utswitch -h enoxec"

Now when I plug my card in the IT system automatically switches me to enoexec, just like it was my local system. A small success.

Tags: Sun Ray solaris

Monday August 08, 2005

Last weeks fun with BT at least had an upside. I got to test both shillix and Sun Ray removable media support. Using a USB key fob I could copy 64M of data to and from the office with blog entries going in and documentation to the Router going out.

So before you put a USB disk into the USB port on the back of your Sun Ray you run this command:

: enoexec.eu FSS 27 $; utdiskadm -l
cannot access parent directories
Device          Partition       Mount Path
------          ---------       ----------
: enoexec.eu FSS 28 $;

Now insert the drive.

: enoexec.eu FSS 28 $; utdiskadm -l
Device          Partition       Mount Path
------          ---------       ----------
disk2           disk2s2         /tmp/SUNWut/mnt/cg13442/z'l_0_1
disk4           disk4s2
: enoexec.eu FSS 29 $; utmount -m disk4s2
: enoexec.eu FSS 30 $; utdiskadm -l
Device          Partition       Mount Path
------          ---------       ----------
disk2           disk2s2         /tmp/SUNWut/mnt/cg13442/z'l_0_1
disk4           disk4s2         /tmp/SUNWut/mnt/cg13442/noname
: enoexec.eu FSS 31 $; ls -l /tmp/SUNWut/mnt/cg13442/noname
total 485
-rwxrwxrwx   1 cg13442  staff      86351 Dec  3  2001 COMMAND.COM
-rwxrwxrwx   1 cg13442  staff      46610 Jan 14  2003 FDISK.EXE
-rwxrwxrwx   1 cg13442  staff      14785 Jan 14  2003 FORMAT.EXE
-rwxrwxrwx   1 cg13442  staff      42107 Aug 26  2002 KERNEL.SYS
-rwxrwxrwx   1 cg13442  staff      57344 Dec 27  2002 Password.exe
: enoexec.eu FSS 32 $; utdiskadm -u /tmp/SUNWut/mnt/cg13442/noname
: enoexec.eu FSS 33 $;

Very cool, well if sneaker net can ever really be cool. Having to manually mount the second drive on the key could just be because we have beta SRSS software on this system. When we upgrade I'll try it again to make sure it works.

Must work out what my bandwidth is when I cycle to work. At only 64Mbytes that would be about 18K/sec if I cand get home in an hour.

Tags: Sun Ray sneakernet