Alan Burlison's Work Related Ramblings

All | General | Java | NetBeans | Perl | Solaris

20080214 Thursday February 14, 2008

Kicking both tyres together - VirtualBox and Indiana

I saw Tim's announcement that Sun were acquiring innotek. innotek provide VirtualBox, which what Tim described as a "high performance type 2 hypervisor". For people who aren't virtualisation wobbleheads, that simply means that it gives you the ability to run operating system on top of another. The host operating system runs on top of the x86 hardware, you run VirtualBox as an application on top of that. VirtualBox emulates the hardware that a guest Operating System needs to run, so that you can run other OSs on top of your host OS. This is in contrast to xVM (Xen), which runs on top of the bare hardware, with the installed OSs running on top of it.

I pulled the Solaris beta binary from the downloads site and had a play with it. The VirtualBox admin GUI is very easy to use, and it was quick and easy to set up a VM to install a copy of Windows XP on top of Solaris - my workstation is running Nevada build 82. Most everything seems to work really well, the only exceptions being sound, which I believe is not implemented yet, and the Solaris/XP file sharing, which seems a little flaky - it works OK for viewing and editing files created on Solaris, however creating new folders from XP works but reports errors, and you can't see the resulting folders in XP. The other nit is that seamless mode doesn't work quite right if you have a dual-monitor (TwinView) setup - details are here.

The other big news this week has been the release of Indiana Preview 2, AKA OpenSolaris Developer Preview 2. Stephen was gently nagging me last week to take a look at Indiana. I didn't want to futz with my machine config as I've just freshly installed build 82 on it, so I decided to see if Indiana2 would run under VirtualBox - and it does.

First you need to make sure you allocate enough memory for Indiana - I set the memory size to 512Mb, and the video memory size to 32Mb. After installing Indiana and rebooting the VirtualBox VM there's a couple of kernel warnings on boot, one about MPO being disabled because the virtualised memory is interleaved and another about their being no randomness provider for /dev/random. The other thing is that Xorg came up by default in 16-bit mode, which caused VirtualBox to complain, although a little tweaking of xorg.conf solved that.

I also figured out how to get the Indiana screen resolution up to 1280x1024 - you need a tweaked xorg.conf, and before you start your VM you need to run VBoxManage setextradata Indiana CustomVideoMode1 1280x1024x32 - replacing "Indiana" with the name of your VM, and note that "CustomVideoMode1" ends in "1" (one) not "l" (L).

The biggest nit is that the virtualised network device used by VirtualBox requires the pcn driver, and it isn't redistributable, which means in turn it isn't part of Indiana. I grabbed a copy from build 82 (the latest SXCE) and installed it into the Indiana image, rebooted and NWAM brought the interface up. If you want to do this yourself, here's the steps:

  1. Get access to the contents of the SXCE ISO image from inside your Indiana VM. Use the VirtualBox Virtual Disk manager (File -> Virtual Disk Manager) to make either the DVD or ISO image available to your Indiana VM.
  2. Extract the files from /Solaris_11/Product/SUNWos86r/archive/none.bz2 - this is a bzip2ed cpio archive.
  3. As root, copy pcn and pcn.conf to /kernel/drv in your Indiana VM.
  4. Run add_drv -i '"pci1022,2000" "pci103c,104c"' pcn to install the driver - note the quotes are important!
  5. Reboot your VM. After rebooting, ifconfig -a should show you now have a working pcn0 network interface.

And of course, if you are worried about brickifying your Indiana VM by doing this, you can always use VirtualBox's snapshot facility to make sure you can roll everything back if something goes wrong.

One important not however: the resulting Indiana VDI image is NOT redistributable, because of licensing restrictions on the pcn driver.

Once I had the Indiana image up and running under Solaris, I copied it onto a Windows XP machine, imported it, attached it to a Solaris VM , started it up and had a preconfigured instance of Indiana running under WinXP, which I thought was kinda neat. I haven't tested it on Linux but I'm sure this would work there too, as well as on OS X. So all you people who have been putting off taking a look at Indiana because it would mean rebuilding your machine now have no excuse :-)

Update - the ae driver

There is an open-source and redistributable alternative to the pcn driver, the ae driver. This is available at http://homepage2.nifty.com/mrym3/taiyodo/ae-2.6.0a.tar.gz#../ae-2.6.0a.tar.gz. The driver tarball contains installation instructions. To make it accessible to your VM, it needs to be made into an ISO image that you can mount into the VM, using the Devices -> Mount CD/DVD-ROM menu of your VM. To make this a little easier I've prebuilt an ISO of the driver tarball which you can download from here. I haven't tried this driver myself, if you use it, let me know how you get on.

Posted by alanbur ( Feb 14 2008, 01:36:58 AM GMT ) Permalink Comments [16]

20071205 Wednesday December 05, 2007

OpenSolaris jobs in Manchester

We've just posted two job adverts for people to come and work on the web infrastructure that runs opensolaris.org These are both based in Manchester, UK, working from Sun's office. If you are interested, please submit a CV here or here. Both roles are identical, so it doesn't matter which link you use :-)

Posted by alanbur ( Dec 05 2007, 02:23:38 PM GMT ) Permalink Comments [0]

20070929 Saturday September 29, 2007

The new opensolaris.org user management webapp moves into the light

As outlined here I've been working away on providing the infrastructure needed to break up the current monolithic J2EE application used to host opensolaris.org. I've just released the source code of the existing portal, a large J2EE application. As detailed in the restructuring document, the intention is to split the functionality of the current webapp into smaller, more manageable parts. As I was releasing the portal code, I thought it was a good time to move my development repository outside as well. Accordingly, you can browse the source at:

http://src.opensolaris.org/source/xref/website/auth/trunk/

and the source can be accessed via anonymous SVN at:

svn+ssh://anon-AT-opensolaris.org/svn/website/auth

It is *very* rudimentary and is *far* from being finished. What I've put on opensolaris.org is just a snapshot of my development workspace as of today. However the database schema is available, along with a small set of dummy data which is used to populate the Derby database currently used by the prototype webapp.If you have any questions, I encourage you to subscribe to the website-discuss mailing list and ask them there.

Posted by alanbur ( Sep 29 2007, 12:48:25 AM BST ) Permalink Comments [3]

Source code for the opensolaris.org portal released

After a long haul, I've finally released the source code of the J2EE web application that runs the opensolaris.org website. If you are interested, the source can be browsed online at:

http://src.opensolaris.org/source/xref/website/portal/trunk/

and the source can be accessed via anonymous SVN at:

svn+ssh://anon-AT-opensolaris.org/svn/website/portal

If you have any questions, I encourage you to subscribe to the website-discuss mailing list and ask them there.

Enjoy!

Posted by alanbur ( Sep 29 2007, 12:41:22 AM BST ) Permalink Comments [0]

20070823 Thursday August 23, 2007

Acrobat on Solaris x86 - a hopeful sign!

The folks over at Adobe have just set up a blog entitled Adobe Reader on Unix, and the first post says:

It's time to get the ball rolling for the much awaited blog for Adobe Reader on Unix platforms. The purpose of this blog is to provide a platform for developers and the users of the product to share ideas, experiences and feedback about the product for the benefit of everyone.

Which is a very hopeful sign - the version of Acrobat which is currently available on Solars x86 is ancient, and I'm certain an up-to-date version would be warmly welcomed all round. Why not head over to the Adbobe blog and give them some encouragement? ;-)

Posted by alanbur ( Aug 23 2007, 02:32:26 PM BST ) Permalink Comments [0]

20061127 Monday November 27, 2006

Hosting a community website using Solaris and Tomcat

Outside of work I'm also a member of a Community-based Samba band, Meninos do Morumbi Oldham, along with my son James.  Like most people in the IT business, once people figure out what you do you inevitably get requests to help with anything related to computers, and I got collared to set up a website for the group.  I started looking around for hosting providers, and my original choice was bluehost.com, based on the recommendation of my colleague Phil.  However, the membership management application I'm also developing is written in Java, which mean I really wanted to run Tomcat on the site as well, and bluehost didn't offer that option.  Finding a hosting provider who would host Tomcat at a reasonable price proved incredibly difficult - the costs were way more than a non-profit like us could afford.  Eventually I stumbled across mod3, a UK-based hosting provider who offer a Solaris 10 zone for the princely sum of £9.95 per year, plus pay-as-you-go for network bandwidth, disk space etc.  The fact that they were running Solaris appealed to me, and I was happy to risk the £10 involved out of my own capacious pocket ;-)  The base hosting package also came with 660Mb of usable disk space, which was plenty for our needs.

After purchasing the site domain name and hosting I started to think about what I wanted to put on the site.  You get 64Mb of RSS memory with a base zone config, which meant I needed to be fairly careful with what I ran on the site.  As I said, I knew I'd probably want use Tomcat at some point, so I began to wonder if I could run the whole thing with Tomcat alone.  From previous experience of setting up similar websites before, I know that setting up the site is relatively easy, the real problem is providing content, then keeping it up-to-date and relevant.  I also didn't want to become the bottleneck for making changes to she site, which pretty much meant that I needed to use some sort of Content Management System, so that I could give other members of the group the ability to edit content.  Nearly everyone in the organisation is a non-IT type (one of the attractions of the group for me ;-), so whatever I used had to make editing easy - requiring that people hack on raw HTML was a non-starter.

As part of the day job I'm looking at the possibility of using a CMS for part of the OpenSolaris website, so I'd already been looking around at what was available (and free!).  I didn't need an 'Enterprise level' CMS - features such as versioning, content staging, multilingual support or workflow management weren't necessary, what I needed was something easy for users to understand, and that didn't have huge resource requirements.  And being written in Java was a bonus, as it meant I could then run it under the Tomcat instance I already knew I was going to need for the membership management stuff.

My final choice was MeshCMS, and I've been extremely happy with it.  It hit all the key requirements that I had - simple to deploy (single WAR file, no database required), easy to customise, structures the site using the directory/subdirectory paradigm that any PC user is already familiar with, has an integrated WYSIWYG editor for editing content, and the clincher - has very modest resource requirements and fits inside my 64Mb RSS constraint.  MeshCMS has been exceedingly well thought out - designing the look and feel of the site involves just modifying a single JSP template, which is then applied to all the pages.  The site navigation menus are all automatically generated from the layout of the directory hierarchy used to store the page content, and most-frequently accessed pages bubble to the top of the menus.  Linking to other pages in the site is easy - the integrated editor provides a dynamically-generated list of pages that you can select from.  The editor even provides a list of the styles defined in the site stylesheet for you to select from - a feature that Roller (the package used to run blogs.sun.com) could well do with emulating.

Having decided on the CMS, the next task was to see if I could minimise Tomcat's footprint as far as possible.  Tomcat runs as a normal Unix process, and so doesn't have permission to open low-numbered ports (below 1024), which is why by default it runs on port 8080.  Webservers normally run on port 80, and using that port requires root privilege.  The Apache webserver gets around this problem by starting up as root, opening port 80 then switching user to the webserver user before starting to serve pages.  However Tomcat is written in Java, so Unix-centric mechanisms like switching user aren't an option.  The normal way to get around this is to put Tomcat behind Apache, using the mod_jk module to shunt traffic between the two.  However because of my memory constraints, I wanted to avoid using Apache if at all possible.  There are a number of fairly vile hacks for doing this for Linux, including stuff such as firewall or userland port redirection, but most of them suffer from various problems.

However because I was running on Solaris, I had a far better option.  I was already intending to run Tomcat as a SMF service, and one of the lesser-known features of SMF is that it is integrated with the Least Privilege mechanism in Solaris 10, which allows you to grant elevated privileges to normal user processes in a controlled way.  This meant that allowing Tomcat to open port 80 simply required granting it the net_privaddr privilege:

    <method_credential user='meninos' group='staff'
      privileges='basic,net_privaddr' />

The really neat thing is that I didn't have to give the meninos user the net_privaddr privilege permanently, it only needs it for the duration of the service start method. The full service manifest is available here if you want to use it yourself.

So, if you are ever in the Greater Manchester area, check out our performances page on the website and if you get a chance, pop by and hear us play ;-)  We were out busking in Manchester yesterday, and for once the weather was kind, despite the time of year :-)

Posted by alanbur ( Nov 27 2006, 10:54:59 AM GMT ) Permalink Comments [5]

20060207 Tuesday February 07, 2006

OpenSolaris buzzes Red Hat

I was reading Roumen's post on the current upsurge in Yahoo search requests ("search buzz") for NetBeans noticed by Tim O'Reilly. The answer to that question is quite simple - NetBeans 5.0 has just been released, and most excellent it is too. However what caught my eye was the last paragraph of Tim's post where he notes that OpenSolaris and Red Hat are neck and neck in the Yahoo search engine popularity stakes. Bearing in mind OpenSolaris has come from zero to neck and neck with Red Hat in such a short time, it appears that Jonathan's predictions weren't that far off base after all.

Posted by alanbur ( Feb 07 2006, 05:49:32 PM GMT ) Permalink Comments [0]

20060125 Wednesday January 25, 2006

OpenSolaris appliances

I see my colleague Chris Gerhard has blogged about the new OpenSolaris Appliances community. Chris's list of requirements is pretty much identical to mine - others have added comments to his blog suggesting a raft of additional stuff, but I'm primarily interested in something that can act as a mail and web server as well as doing SMB filesharing. My constraints are that it must be small, quiet and run Solaris (natch!), so if you have any suggestions for where we should start on the hardware front, please head over to the forum and help us get started. I've disabled comments to encourage you to post over there and not here ;-)

Posted by alanbur ( Jan 25 2006, 05:28:05 PM GMT ) Permalink

20060109 Monday January 09, 2006

An opensolaris.org hack: customisable RSS feeds

In a (rather futile!) attempt to keep my email volume down slightly I've taken to reading some of the OpenSolaris mailing lists / forums via the RSS feeds provided by the site. I use the most excellent Sage plugin for Firefox. You can find a list of all the available opensolaris.org RSS feeds here - there are four (!) variants for each OpenSolaris forum, plus roll-up feeds for each group of forums. Be aware that there's a glitch on the webpage and the feed type icons are mislabelled - the "Threads" (one big speech bubble) and "Messages" (two small speech bubbles) labels are transposed, although I'm sure Derek will have that sorted out shortly - if you hover over the icons in the feed list, the "Threads" URLs all contain rssthreads.jspa and the "Messages" links contain rssmessages.jspa.

One of my favorite feeds is the popular threads feed, which shows you the threads with the most activity. I also subscribe to a few individual forum feeds, such as the OpenSolaris discuss one, but something that has always niggled me is that you only see the last 10 threads, which really isn't enough. However it is possible to change this by modifying the bookmark you use for the feed - simply stick &numItems=20 on the end of the bookmark URL. The full list of options are called out in the Jive documentation. Unfortunately, despite what the documentation suggests this doesn't work for "popular threads" feeds, which is a shame.

Posted by alanbur ( Jan 09 2006, 11:07:19 AM GMT ) Permalink Comments [0]

20041007 Thursday October 07, 2004

Hacking loadable kernel modules with perl

I was reading Alan Coopersmith's blog entry on how he hacked the Xserver with the aid of perl, and that reminded me of a hack I did myself a while back, using perl.

Here's the scenario: I work mainly from home, using a Solaris 10 machine running a debug kernel and VPN to connect to work. The VPN connection requires the use of a loadable STREAMS driver supplied by Cisco. I also run Gnome, but one shortcoming of gnome-term is it doesn't support /dev/console. To get around this I have a single dtterm started with the command-line dtterm -C -map -title Console, which means that any console output is captured by the dtterm. Normally it's iconified, but the -map flag means that if anything is output to /dev/console it de-iconifies so I can see the message.

Here's the rub: If you run a debug kernel you get a kernel thread that periodically tries to unload any loadable kernel modules. This is to flush out any problems with modules that don't load and unload properly, for example by leaking memory. The Cisco VPN module locks itself into memory (which is sort of OK), but every time it gets an unload request it logs the fact with the following cmn_err() message which ends up on /dev/console, which definitely isn't OK. 

Oct  6 18:18:06 myhost vpnmod: VPN Module 5.2.4 Unload BUSY (Wed Mar 13 00:31:06 MST 2002)

Every time this happens my /dev/console dtterm de-iconifies and I end up playing whack-a-mole.

OK, let's nail that sucker. It's complicated by the fact that we don't have source for the Cisco VPN module, but who's Operating System it it anyway, right? ;-) First, let's find the culprit: 

# modinfo | grep vpn
100 7b600000  1f650   -   1  vpnmod (VPN module)
# ifconfig bge0 modlist
0 arp
1 ip
2 vpnmod
3 bge
# find /kernel -name vpnmod
/kernel/strmod/sparcv9/vpnmod
/kernel/strmod/vpnmod
# cp /kernel/strmod/sparcv9/vpnmod vpnmod.sparc.64
# elfdump -s vpnmod.sparc.64 | egrep 'cmn_err|print'
     [178]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       vsprintf
     [193]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       vcmn_err
     [255]  0x000000000000 0x000000000000  NOTY GLOB  D    0 UNDEF       cmn_err

So there's the potential culprits. We know vpnmod is a STREAMs driver, and that it feels the need to tell us whenever it gets an unload request, so let's try to narrow down where that might be. If we look at the Device Driver Entry Points section of the 'Writing Device Drivers' manual, we can see that all loadable modules (including STREAMS drivers) have to implement the _init, _fini and _info entry points, and that the _fini entry point is called when a module is being unloaded, so that looks like the obvious place to start. 

#  dis -F _fini vpnmod.sparc.64 | less
                ****   DISASSEMBLER  ****


disassembly for vpnmod.sparc.64

section .text
_fini()
    _init+0x120:            9d e3 bf 30  save         %sp, -0xd0, %sp
    _init+0x124:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x128:            90 12 20 00  or           %o0, vpn_open, %o0        ! vpn_open
    _init+0x12c:            91 2a 30 20  sllx         %o0, 32, %o0
    _init+0x130:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x134:            90 02 00 09  add          %o0, %o1, %o0
    _init+0x138:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x13c:            40 00 00 00  call         _init+0x13c
    _init+0x140:            01 00 00 00  nop          
    _init+0x144:            d0 27 a7 eb  st           %o0, [%fp + 0x7eb]
    _init+0x148:            d0 07 a7 eb  ld           [%fp + 0x7eb], %o0
    _init+0x14c:            80 a2 20 00  cmp          %o0, 0x0
    _init+0x150:            12 48 00 13  bne,pt       %icc, _init+0x19c
    _init+0x154:            01 00 00 00  nop          
    _init+0x158:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x15c:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x160:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x164:            92 12 60 00  or           %o1, 0x0, %o1
    _init+0x168:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x16c:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x170:            91 2a 30 20  sllx         %o0, 32, %o0
    _init+0x174:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x178:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x17c:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x180:            92 02 40 08  add          %o1, %o0, %o1
    _init+0x184:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x188:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x18c:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x190:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x194:            10 68 00 11  ba,pt        %xcc, _init+0x1d8
    _init+0x198:            01 00 00 00  nop          
    _init+0x19c:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x1a0:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]

    _init+0x1a4:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x1a8:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x1ac:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x1b0:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x1b4:            93 2a 70 20  sllx         %o1, 32, %o1
    _init+0x1b8:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x1bc:            11 00 00 00  sethi        %hi(0x0), %o0
    _init+0x1c0:            d2 5f a7 df  ldx          [%fp + 0x7df], %o1
    _init+0x1c4:            92 02 40 08  add          %o1, %o0, %o1
    _init+0x1c8:            d2 77 a7 df  stx          %o1, [%fp + 0x7df]
    _init+0x1cc:            d0 5f a7 df  ldx          [%fp + 0x7df], %o0
    _init+0x1d0:            90 12 20 00  or           %o0, 0x0, %o0
    _init+0x1d4:            d0 77 a7 df  stx          %o0, [%fp + 0x7df]
    _init+0x1d8:            90 10 20 00  clr          %o0
    _init+0x1dc:            13 00 00 00  sethi        %hi(0x0), %o1
    _init+0x1e0:            92 12 60 00  or           %o1, vpn_open, %o1        ! vpn_open
    _init+0x1e4:            93 2a 70 20  sllx         %o1, 32, %o1
    _init+0x1e8:            15 00 00 00  sethi        %hi(0x0), %o2
    _init+0x1ec:            92 02 40 0a  add          %o1, %o2, %o1
    _init+0x1f0:            92 12 60 00  or           %o1, 0x0, %o1
    _init+0x1f4:            15 00 00 00  sethi        %hi(0x0), %o2
    _init+0x1f8:            94 12 a0 00  or           %o2, vpn_open, %o2        ! vpn_open
    _init+0x1fc:            95 2a b0 20  sllx         %o2, 32, %o2
    _init+0x200:            17 00 00 00  sethi        %hi(0x0), %o3
    _init+0x204:            94 02 80 0b  add          %o2, %o3, %o2
    _init+0x208:            94 12 a0 00  or           %o2, 0x0, %o2
    _init+0x20c:            d6 5f a7 df  ldx          [%fp + 0x7df], %o3
    _init+0x210:            19 00 00 00  sethi        %hi(0x0), %o4
    _init+0x214:            98 13 20 00  or           %o4, vpn_open, %o4        ! vpn_open
    _init+0x218:            99 2b 30 20  sllx         %o4, 32, %o4
    _init+0x21c:            1b 00 00 00  sethi        %hi(0x0), %o5
    _init+0x220:            98 03 00 0d  add          %o4, %o5, %o4
    _init+0x224:            98 13 20 00  or           %o4, 0x0, %o4
    _init+0x228:            40 00 00 00  call         _init+0x228
    _init+0x22c:            01 00 00 00  nop          
    _init+0x230:            d0 47 a7 eb  ldsw         [%fp + 0x7eb], %o0
    _init+0x234:            b0 10 00 08  mov          %o0, %i0
    _init+0x238:            81 cf e0 08  return       %i7 + 0x8
    _init+0x23c:            01 00 00 00  nop

Hmm, wait a minute, there doesn't apppear to be any calls to cmn_err and friends in there, and if we look carefully at the call instructions we notice that the calls all call the location of the call instruction itself - what on earth is happening? Well, loadable modules are more-or-less standard ELF files, and when they are loaded into the kernel they undergo the same kind of link editing as userland ELF files get from ld.so.1. What we actually need to do is to look at the function after relocation, and the easiest way to do that is to look on a running system using mdb: 

# mdb -k
Loading modules: [ unix krtld genunix specfs ufs sd ip sctp usba s1394 nca
crypto random nfs audiosup ptm ipc ]
> vpnmod`_fini ::dis
vpnmod`_fini:                   save      %sp, -0xd0, %sp
vpnmod`_fini+4:                 sethi     %hi(0), %o0
vpnmod`_fini+8:                 or        %o0, 0, %o0
vpnmod`_fini+0xc:               sllx      %o0, 0x20, %o0
vpnmod`_fini+0x10:              sethi     %hi(0x70150000), %o1
vpnmod`_fini+0x14:              add       %o0, %o1, %o0
vpnmod`_fini+0x18:              or        %o0, 0x130, %o0
vpnmod`_fini+0x1c:              call      -0x7a3d10b8   <mod_remove>
vpnmod`_fini+0x20:              nop
vpnmod`_fini+0x24:              st        %o0, [%fp + 0x7eb]
vpnmod`_fini+0x28:              ld        [%fp + 0x7eb], %o0
vpnmod`_fini+0x2c:              cmp       %o0, 0
vpnmod`_fini+0x30:              bne,pt    %icc, +0x4c   <vpnmod`_fini+0x7c>
vpnmod`_fini+0x34:              nop
vpnmod`_fini+0x38:              sethi     %hi(0), %o0
vpnmod`_fini+0x3c:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x40:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x44:              or        %o1, 0, %o1
vpnmod`_fini+0x48:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x4c:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x50:              sllx      %o0, 0x20, %o0
vpnmod`_fini+0x54:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x58:              sethi     %hi(0x7b61c800), %o0
vpnmod`_fini+0x5c:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x60:              add       %o1, %o0, %o1
vpnmod`_fini+0x64:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x68:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x6c:              or        %o0, 0x60, %o0
vpnmod`_fini+0x70:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x74:              ba,pt     %xcc, +0x44   
vpnmod`_fini+0x78:              nop
vpnmod`_fini+0x7c:              sethi     %hi(0), %o1
vpnmod`_fini+0x80:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x84:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0x88:              or        %o0, 0, %o0
vpnmod`_fini+0x8c:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0x90:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0x94:              sllx      %o1, 0x20, %o1
vpnmod`_fini+0x98:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0x9c:              sethi     %hi(0x7b61c800), %o0
vpnmod`_fini+0xa0:              ldx       [%fp + 0x7df], %o1
vpnmod`_fini+0xa4:              add       %o1, %o0, %o1
vpnmod`_fini+0xa8:              stx       %o1, [%fp + 0x7df]
vpnmod`_fini+0xac:              ldx       [%fp + 0x7df], %o0
vpnmod`_fini+0xb0:              or        %o0, 0x98, %o0
vpnmod`_fini+0xb4:              stx       %o0, [%fp + 0x7df]
vpnmod`_fini+0xb8:              clr       %o0
vpnmod`_fini+0xbc:              sethi     %hi(0), %o1
vpnmod`_fini+0xc0:              or        %o1, 0, %o1
vpnmod`_fini+0xc4:              sllx      %o1, 0x20, %o1
vpnmod`_fini+0xc8:              sethi     %hi(0x7b61c800), %o2
vpnmod`_fini+0xcc:              add       %o1, %o2, %o1
vpnmod`_fini+0xd0:              or        %o1, 0xa0, %o1
vpnmod`_fini+0xd4:              sethi     %hi(0), %o2
vpnmod`_fini+0xd8:              or        %o2, 0, %o2
vpnmod`_fini+0xdc:              sllx      %o2, 0x20, %o2
vpnmod`_fini+0xe0:              sethi     %hi(0x7b61c800), %o3
vpnmod`_fini+0xe4:              add       %o2, %o3, %o2
vpnmod`_fini+0xe8:              or        %o2, 0x90, %o2
vpnmod`_fini+0xec:              ldx       [%fp + 0x7df], %o3
vpnmod`_fini+0xf0:              sethi     %hi(0), %o4
vpnmod`_fini+0xf4:              or        %o4, 0, %o4
vpnmod`_fini+0xf8:              sllx      %o4, 0x20, %o4
vpnmod`_fini+0xfc:              sethi     %hi(0x7b61d400), %o5
vpnmod`_fini+0x100:             add       %o4, %o5, %o4
vpnmod`_fini+0x104:             or        %o4, 0x3d0, %o4
vpnmod`_fini+0x108:             call      -0x7a4a2388   <cmn_err>
vpnmod`_fini+0x10c:             nop
vpnmod`_fini+0x110:             ldsw      [%fp + 0x7eb], %o0
vpnmod`_fini+0x114:             mov       %o0, %i0
vpnmod`_fini+0x118:             return    %i7 + 8
vpnmod`_fini+0x11c:             nop

Ahah! At vpnmod`_fini+0x108 there's a call to cmn_err. If we look a the cmn_err manpage, we see the prototype is void cmn_err(int level, char *format...);, so the format string will be passed in the %o2 register, this being sparc. Looking back through the disassembly, we can work out that %o2 will have the value 0x7b61c8a0 at the time of the call to cmn_err. Back to mdb: 

> 0x7b61c8a0/S
0x7b61c8a0:     VPN Module %s Unload %s (%s)\n

Bingo. That matches the string we are seeing printed to /dev/console, so we have identified the offending call, all we now have to do is to get rid of it. The obvious thing is to replace the call with a nop instruction. We can calculate the offset of the offending instruction fairly easily: we know the offset fom the beginning of the _fini routine, it's 0x108, and we can use dump to tell us the start address of the routine: 

# dump -t -n _fini vpnmod.sparc.64

vpnmod.sparc.64:

              ***** SYMBOL TABLE INFORMATION *****

[Index]  Value           Size         Type	Bind	Other	Shndx	Name
.symtab:
[252]	 0x1ca4          115724       2		1	0	0x1	_fini

So that puts the offending call instruction at 0x1ca4 + 0x108 or 0x1dac from the start of the vpnmod code segment. Let's just double-check we are right with mdb (output shortened for clarity): 

> ::modinfo
 ID         LOADADDR     SIZE REV MODULE NAME
:
100         7b600000    1f650   1 vpnmod (VPN module)
:
> 7b600000+0x1dac ::dis
:
vpnmod`_fini+0x108:             call      -0x7a4a2388   <cmn_err>
:
>

Yep, that works. However, an ELF file such as vpnmod isn't a straight memory image, so we can't just seek that number of bytes into the file and write a nop in there, we need to make sure it's the correct offset from the start of the segment that holds the code - the .text segment. We can find that bit of information out as follows: 

# dump -hvn .text vpnmod.sparc.64  

vpnmod.sparc.64:

	   **** SECTION HEADER TABLE ****
[No]	Type	Flags	Addr           Offset         Size            Name
	Link	Info	Adralgn        Entsize

[1]	PBIT    -AI	0              0x40           0x1c40c         .text
	0	0	0x8            0

So that's telling us that the .text (code) segment of the ELF file is 0x40 bytes from the start of the file, which means that we can finally calculate the position within the ELF file of the instruction we want to nop out - <offset of .text section in file> + <offset of _fini routine from start of .text segment> + <offset of call instruction from start of _fini> = 0x40 + 0x1ca4 + 0x108 = 0x1dec. Yay! A we have to do is splat a nop over the 4 bytes starting at that position and we are done, right?

No, unfortunately not. Remember I said that the kernel link-edits the module as it loads it in? Well, if we just nop the offending instruction krtld (the kernel linker) will still attempt to relocate the call to cmn_err, specifically it assumes the first byte is already a call opcode (0x40) and it will write the offset to cmn_err into the 2nd, 3rd and 4th bytes of the call instruction - only it won't be a valid call instruction any more as we've just written a nop over the top of it. Damn.

What we need to do is to prevent the kernel performing the relocation for the call that we wish to nop out. How on earth do we do that? Well, let's poke around inside the ELF file some more. We can use elfdump to dump out the relocation information in the file, and we already know that the relocation we are looking for is for a call to cmn_err that is 0x1dac bytes from the start of the .text segment. 

# elfdump -r vpnmod.sparc.64 | less           

Relocation Section:  .rela.text
        type                          offset     addend  section        with respect to
:
        R_SPARC_WDISP30               0x1c8c          0  .rela.text     cmn_err  
        R_SPARC_WDISP30               0x1dac          0  .rela.text     cmn_err  
        R_SPARC_WDISP30               0x1e18          0  .rela.text     cmn_err  
:

There it is. A quick read of the libgelf manpage reveals the following information: 

     gelf_getrela()
           Retrieves the  Elf32_Rela  or  Elf64_Rela  information
           from  the  relocation  table  at  the given index. dst
           points to the location where the GElf_Rela  relocation
           entry will be stored.

So it's off to /usr/include to find the definition of Elf64_Rela: 

typedef struct {
	Elf64_Addr	r_offset;
	Elf64_Xword	r_info;		/* sym, type: ELF64_R_... */
	Elf64_Sxword	r_addend;
} Elf64_Rela;

OK, let's see if we can persuade the linker to change what it does with the relocation that we want to nobble. Searching for the existing relocation type, R_SPARC_WDISP30, in the /usr/include header files reveals this: 

:
#define R_SPARC_NONE		0		/* relocation type */
#define R_SPARC_8		1
#define R_SPARC_16		2
#define R_SPARC_32		3
#define R_SPARC_DISP8		4
#define R_SPARC_DISP16		5
#define R_SPARC_DISP32		6
#define R_SPARC_WDISP30		7
:

Hmm, that R_SPARC_NONE sure looks interesting. At this point we suspect that the mighty Linker and Libraries Guide may have some words of wisdom - although it documents the workings of ld and ld.so.1, much of the information on how ELF files work is also applicable to the kernel. Sure enough, Chapter 7 on Object File Format has a strong hint that R_SPARC_NONE relocations are ignored by the linker, so if we set the r_info field of the appropriate Elf64_Rela entry in the .rela.text section to R_SPARC_NONE we should be able to get the kernel linker to ignore the relocation for the call we wish to nop out. At this point, I went off and looked at the source of the krtld, the kernel linker to confirm that this was in fact what happened - an option that once Solaris is released as Open Source will be open to you as well :-)

Our story is almost told - the process of overwriting the appropriate relocation table section is much the same as that used for nop-ing out the function call - find the start of the .rela.text table in the ELF file, find the index in the .rela.text table of the entry we want to null out, write zeros over it, and we're done.

Those of you who have lasted this far are probably asking "What on earth has this got to do with perl?". Well, there are some slight differences between the way this has to be done on 32-bit sparc, 64-bit sparc and i386, and I needed fixed versions of the VPN module for all those platforms. The vpnmod module is also unnecessarily chatty when it's loaded up during boot, and I wanted to shut it up then as well. I therefore wrote a perl script to do all this automatically. Unfortunately we aren't permitted to post source code on our blogs so I can't share it with you, but I can let you see it in action: 

# patchmod _fini+0x108 vpnmod.sparc.64 vpnmod.sparc.64.patched
.text segment base is 0x40, length is 0x1c40c
.rela.text segment base is 0x21808, length is 0x11880,
    each entry is 0x18 bytes long
function _fini base is 0x1ca4, length is 0x1c40c
call opcode is at offset 0x1dac in the .text segment
call opcode is at offset 0x1dec in the file
relocation for address 0x1dac is at index 363 in .rela.text,
    and is a call to cmn_err
relocation table entry is at offset 0x23a10 in the file
copying and patching file
done

And if we dig around with dis and dump we can see that the script has had the desired effect: 

# dis -F _fini vpnmod.sparc.64.patched | less
:
    _init+0x218:            99 2b 30 20  sllx         %o4, 32, %o4
    _init+0x21c:            1b 00 00 00  sethi        %hi(0x0), %o5
    _init+0x220:            98 03 00 0d  add          %o4, %o5, %o4
    _init+0x224:            98 13 20 00  or           %o4, 0x0, %o4
    _init+0x228:            01 00 00 00  nop          
    _init+0x22c:            01 00 00 00  nop          
    _init+0x230:            d0 47 a7 eb  ldsw         [%fp + 0x7eb], %o0
    _init+0x234:            b0 10 00 08  mov          %o0, %i0
    _init+0x238:            81 cf e0 08  return       %i7 + 0x8
:
# dump -rvp -n .rela.text vpnmod.sparc.64.patched | less
:
0x1c8c		cmn_err			R_SPARC_WDISP30	  0
0		0			R_SPARC_NONE	  0
0x1e18		cmn_err			R_SPARC_WDISP30	  0
:

It's worth noting that I could have done this all in C - Solaris has libraries for directly manipulating ELF files, see the gelf(3ELF) and elf(3ELF) manpages, but hey, I was in a hurry ;-) Posted by alanbur ( Oct 07 2004, 03:06:40 PM BST ) Permalink Comments [0]