Mike Ramchand's blog, covering mainly JET (Jumpstart Enterprise Toolkit), but with diversions into other stuff, like sailing.. JET Stream

I was giving my standard pitch covering Sun Virtualisation Technologies which essentially entails me talking at length about the following slide. I kind of make the point that it is a means not an end, and that really what you're trying to do is run more applications on the same piece of hardware, and the trick is to try and do it most efficiently.

I've done this talk countless times with a pretty wide variety of customers, and I tend to start talking from the right, and move to the left, explaining how the "layer of additional non app resource utilisation" grows from right to left. (not true for Dynamic System Domains on the far left though.)

It finally occurred to me yesterday that the Unix and Windows world have an utterly different view of Virtualisation, and the need for Virtual Machines.

In the Unix world, application co-existence is second nature to most sys-admins. Virtualisation is all about application isolation. i.e. You START with all the apps on a single O/S instance, and you apply more and more constraints: resource management --> containers/zones --> virtual machines --> dynamic system domains.

In the Windows world, application co-existence is so rare that it isn't even considered. Virtualisation is all about application consolidation. (and the primary way to do it is to create lots of virtual machines, one per app).

The problem is simply viewed from completely different angles.

It's not just an application co-existence issue either. There's a scalability problem. A lot of "other" O/S's can't scale to fit the larger more powerful Intel/AMD multi-core servers available today, and the ONLY way to make use of them is to carve them up into smaller machines that Windows or Linux can digest. Solaris, on the other hand can scale up to hundreds of CPUs, and hundreds of Gigabytes of memory.

From the slide above, Sun/Solaris provides a wide choice of technologies to allow multiple applications to be consolidated onto a single server. Don't let the limitations of "other" O/S's blinker you into thinking that a Virtual Machine is the only answer. 

(I'm not saying that Virtual Machines is the WRONG answer btw, it's ONE of the answers, but it shouldn't be the only one that is considered.)

tags: virtualization containers ldom zones vmware xen

 So, I've been working on JET 4.7 for a while now. A couple of weeks ago I released it internally to Sun, and no-one's complained about any showstopping bugs yet, so I've released a fresh bundle to the sun.com downloads site. The best way to go straight to it is to follow the link at the top of the wiki site.

JET 4.7 Changes:

The MAJOR JET change is that I've kind of merged JetNAS functionality into the main SUNWjet module. This isn't an exact replication of the JetNAS code, and it does things a little differently, but in short you can choose to place the NFS functionality somewhere different from your JET server.

Other JET changes are covered by bugids: 6835016, 685027, 685023, 685019, 685030.

In summary:

6835016:  JET needs to support alternative servers for NFS media. The JET User Guide has already been updated to cover this functionality. Additionally, I've written a short article about it on the JET wiki site.

685027:  JET jetjump.xml script does not stop graphical login from starting. Tidied it up. graphical login no longer starts until J/S is complete.

685023: JET should not create /dhcp directory. We used to do this to shorten the dhcp macro. This is no longer necessary, so we've removed the complication of having it.

685019: JET ksh scripts do not work when using OpenSolaris. There were a couple of constructs in our ksh code that ksh93 choked on. We've updated them so that they are compatible with ksh and ksh93. This allows all you OpenSolaris people to install JET on your OpenSolaris based Jumpstart server. (Note, AI is required to "jumpstart" OpenSolaris.)

685030: JET error message when suitable server is not found could be more meaningful. Well, this is just an attempt to reduce the e-mail traffic. Probably the first error, people new to JET see. We've updated the error message to tell you what to do to fix it.

JET 4.6 - 4.7 Change Summary:

For all you "external" people, the last externally available version was 4.6. Here are the changes from 4.6 - 4.7:

4.6.1: Ability to compress the rpool, and specify /var for zfs boot builds.

4.6.2: 6798648: JET needs to workaround Bug: 6528699: This is to stop Jumpstart from labelling ALL your disks and blowing away all your data. Also added a tweak to the ssh scripts to generate the keys properly.

4.6.3: Added an additional bootadm update-archive to the end of the finish.common script. We were getting some situations where the first reboot had a stale boot-archive, possibly due to a race condition in SPARC S10 Update 6 and above builds.

JET bundled Modules:

We've updated some/most of the bundled products. ALL of them (if required) have been updated to make them ksh93 compatible. Some of them have had more substantial changes:

JetLDOM

This module has had the most substantial rewrite. We've gone against ALL JET precedent and completely changed the structure of the ldom.conf file, so it's not backward compatible (for once). We needed to do that so that we could provide a much easier mechanism to make use of the ldm 1.3 functionality. Sorry.... but you'll like the new one better.

JetFLASH

I've updated JetFLASH so that is actually checks the flar (if it can), to see that it is the right architecture, and the right boot filesystem for the build. (Yes, zfs based flash is coming, but it is a different format to the ufs based one). Bruce added some code to clean up service tags if required.

JetZFS

Quite a few changes in this module. You can now update your rpool properties, I've provided more verbose progress messages (so you can see exactly what it is doing), added code to deal with converting EFI disks back to SMI (it does it for you if required), and finally added a strange hack so that you can optionally use ::: instead of = in your variables to get around a xVM OC gui restriction.

JetZONES

Fixed a bug where make_template was in the wrong place, and added some code to boot the zone before trying to add stuff to the rootpath/var dir. (because it might not be there unless the zone is booted). (Thanks to Mike Hendon for pointing it out). (plus a bunch of new functionality like support for branded zones)

Summary

So just a reminder, the following modules have NEW template variables. Old templates (other than the ldom based ones) will continue to work, but if you want to make use of the new functionality (with the new comments that describe them), you'll need to refresh your templates. (Have a scan through sample.template in /opt/SUNWjet/Templates to see what's changed.)

tags: Solaris JET Jumpstart

I've been working on JET 4.7 (current released JET version is 4.6.3).

The main change is that I've now provided support to allow the Solaris Media and the /opt/SUNWjet directories to be hosted by a server (or servers) other than the JET server. This effectively removes the requirement for the JET server to provide NFS services.

Along the way I was able to tidy up and consolidate a lot of code, particularly the client_allocation_* methods. I've added a few functions, and used the variables more consistently throughout. For those of you doing DHCP/PXE based builds, I've removed the /dhcp hack that was used to overcome the 255 char limitiation of the DHCP macros.

Full details on the wiki site

I'm hoping to release it by the end of April.

tags: Solaris JET Jumpstart

A few months ago I wrote  Solaris Cluster on a laptop using VirtualBox, iSCSI and a quorum server, which detailed some of the hoops that I needed to jump through to get it to work. The good news is that the new features in VirtualBox 2.0 (and now 2.1) have gotten rid of 2 of those hoops:

1.  I no longer need to play with vnics and that crazy script as Host Interface networking simply works out of the box
2. I don't need to install the 32bit internal only SC package, as VirtualBox now supports 64bit guests.

I'm in the process of rebuilding my laptop cluster making use of the above new features.

 Hurrah!!!

tags: JET Jumpstart SunCluster Solaris VirtualBox xVM

As a follow-on to my previous post, JET 4.6 is now available externally. Check the JET wiki which has a link to the download.

This is mainly a bug fix release, and has been given a major number increase because it's the JET version that will be in the next release of xVM OpsCenter. (and we like to release the OpsCenter releases at major version numbers).

In terms of changes, I've fixed a couple of bugs with turning ntp on, and multiple disks in zfs pools for ZFS disks. This version also has tweaks in place to workaround some DHCP issues when installing S10 U6 and Solaris Nevada.

tags: Solaris JET Jumpstart

As a follow-on to my previous post, JET 4.4.7 is now available externally. Check the JET wiki which has a link to the download.

Also have a look at the JET User Guide which has now been posted on the wiki. I expect this to grow over time, but its a conversion from the original user guide with updates for correctness.

tags: Solaris JET Jumpstart

I was in the process of playing with some stuff using VirtualBox, and it occurred to me that this was a pretty cool screenshot, so I did, and here it is.

tags: Solaris belenix ubuntu opensuse virtualbox

Over the past few weeks/months, I've been involved (sometimes peripherally) with setting up our demo kit in the UK Customer Briefing Centre. One of my design goals was to ensure seamless access to our demos from anywhere in the world or more realistically anywhere in the UK, and more specifically, at any customer site. 

Within the CBC network (which is not connected to our internal Sun network) we set up a Sunray Server also running SGD. We initially had some hiccups getting the firewall traversal to work, and had to resort to using VNC which was adequate, but painful as we were tunnelling through ssh, so there were a bunch of steps required to ssh in, start vncserver, then re-ssh in to tunnel the right port, then start vncviewer and after all of that the screen refresh rate wasn't stellar.

However, once we got SGD working it was absolutely awesome.

If you want to have a play with how it works, simply go to Sun's SDG demo site . Unfortunately MyDesktop doesn't work (for obvious reasons) for anonymous users, but it will let you play with the webtop functionality.

For our purposes, we used the MyDesktop functionality. Running a desktop session inside the CBC from anywhere in the world was as easy as pointing a browser at the right place, logging in and clicking on the Desktop icon. Within seconds, as if by magic, a full screen desktop takes over your current screen (you can switch back to local using Alt-TAB). I'd show you a picture of it, but it just looks like a normal desktop and feels remarkably snappy even if it is remote.

The reason we were doing all of this is that we  (The UK Systems Practice) have been running some "Discovery Days" around LDOMS and ZFS which have live demos of the technology. The cool part is that we get to showcase the SGD technology at the same time!

tags: Solaris SGD ZFS LDoms

I've made some minor modifications to share_isos. I realised that add_install_client creates entries in /etc/dfs/dfstab for the "boot" filesystem if not already mounted. Unfortunately, it only checks the dfstab, rather than checking the actual shares. The result is that I need to populate the /etc/dfs/dfstab file.

The complication is that now that it's in dfs, the stubs get shared when the server boots, so I need to unshare them before doing the lofi mounts.

 I've updated the share_isos script to do that. I've decided I'm simply going to make it an add_on JET module that can be installed if that feature is required/desired. (Look out for a JetISO module any day soon).

Finally, while testing all of this I decided to so something pretty obscure. I'm in the process of building a JET.vdi and I'm trying to make the .vdi as small as possible. (A virtual disk image of the smallest Solaris I can use, with JET and its modules installed so that it is nice and easy to distribute (so far its about 800k). I obviously don't want to prepopulate with any Solaris media as that will simply make my vdi too large. The cool thing about iso images is that I can NFS mount a directory containing them, and then I can lofiadm and mount them locally.

So I ended up with a JET server running in a virtualbox, NFS mounting a Sol 10 0508 iso from my home server, lofimounted and shared with share_isos, and I used that to jumpstart another Solaris virtualbox machine on the same PC (which was running XP). The really good news was that it worked. The not so good news is that it took over a day to install. I think there are a few too many levels of indirection and virtualisation between the iso and the virtual disk.

I should really do a bit more testing and figure out the bottleneck. Ideally I'd prefer to keep the isos on the host and use shared folders to allow the virtual JET server to access them, but that won't be possible if the performance is as bad as my original test.

tags: Solaris Jumpstart JET virtualbox

I was intrigued by Gordon Johnston's blog article about creating a 2 node Cluster Express cluster using VirtualBox, and I decided I had to try it.

Here's the final result:

(You can click on the image for a fullsized picture.)

It shows a VNC connection to my laptop which has 4 windows up: The 2 VirtualBox instances showing cxnode1 and cxnode2, a terminal window on the laptop itself, showing the quorum configuration and the iSCSI share, and in the background, the VirtualBox console showing the network configuration of cxnode1. Finally, in the main screen, I'm running Sun Cluster Manager and connected to the cluster, showing the topology and the Resource Group configuration of apache.

Rather than simply duplicating what Gordon did, I decided to go with Solaris Cluster 3.2 U1 as my cluster software. This meant that I had to use an internally available package of 32 bit kernel drivers, as SC 3.2 is shipped as a 64bit only product. (Cluster Express has both 32 and 64 bit variants.) The reason for going with Solaris Cluster is that there are instances where I want to duplicate a scenario that a customer is seeing, and in most cases, that's on Solaris Cluster.

I'm hesitant to duplicate Gordon's article, as I pretty much followed his guide, so I urge you to read his article as it is quite detailed, I'll only highlight where I changed things.

1.  I only really wanted a working cluster, went for the simpler set up of HA Apache
2. Cluster node O/Ss would be Solaris 10 05/08
3. Host O/S is Solaris Nevada Build 93 (as that was what was already running on my laptop)
4. I wanted to Jumpstart the nodes using JET.

Preparing your physical host

I already had VirtualBox 1.6.2 installed, so there wasn't much to do there. The main trick was configuring my laptop to allow Host Interface networking. This bit is Solaris specific. You basically need to do 2 things.

Firstly, you need to set up a bunch of virtual nics, and assign MAC addresses to them. As this needs to be redone on every reboot, I followed Dave Tong's advice  and set up an rc3.d script. (and no, I didn't bother to use SMF either, you can bite me too if you want.)

#!/sbin/sh
state="$1"
case "${state}" in
'start')
    echo 'Plumbing VNICs'
    for MAC in 10 20 30 40
    do
        VNIC=`/usr/lib/vna nge0 c0:ff:ee:0:0:$MAC`
        ifconfig $VNIC plumb
    done
    ;;
'stop')
    ;;
*)
   echo "Usage: $0 { start | stop }"
    exit 1
    ;;
esac

(I did use Gordon's cute fake MAC address c0:ff:ee 'cos its easy to remember)

Secondly, you REALLY REALLY need to make sure that you give the VBoxSVC privileges to configure the interface, otherwis your virtual machines will fail when starting. Whenever you start VirtualBox, you need to get the PID of the VBoxSVC, and run: ppriv -s AEI+net_rawaccess <pid>

Creating your Virtual hosts

Basically I followed Gordon's instructions for this, with TWO exceptions. I needed to select Network in the Boot Order menu in Settings-->General-->Advanced as PXE is not enabled by default. I also found that I couldn't do a PXE boot if I used the Intel driver, so I used the PC-net FAST III driver throughout which worked fine.

Installing Solaris on the Virtual hosts

I've already got JET installed on my laptop, so preparing to install Solaris was an amazingly simple process:

1. /opt/SUNWjet/bin/make_template cxnode1 base_config
2. Edit the template
3. /opt/SUNWjet/bin/make_client cxnode1
4. Start the cxnode1 Virtual Machine, Press F12, Select Lan Boot, and watch it install.
5. /opt/SUNWjet/bin/make_template -T cxnode1 cxnode2
6. Edit the cxnode2 template and change 3 variables.
7. /opt/SUNWjet/bin/make_client cxnode2
8. Start the cxnode2 Virtual Machine. Press F12, Select Lan Boot and watch it install.
   

Of course, the devil is in the detail, I'd already configured the Solaris media, and these are the changes I made to a default template for the build to work:

base_config_ClientArch=i86pc
*base_config_ClientEther=c0:ff:ee:0:0:10
base_config_ClientOS=10_0508x
base_config_client_allocation="dhcp"
*base_config_sysidcfg_ip_address=192.168.1.201
base_config_sysidcfg_netmask=255.255.255.0
base_config_profile_fdisk="alldisk"
base_config_profile_cluster=SUNWCXall
base_config_profile_root=free 
base_config_profile_swap=2048
base_config_profile_s6_mtpt="/globaldevices"
base_config_profile_s6_
base_config_ipmp_networkifs="pcn0_pcn1"
*base_config_ipmp_networkif_pcn0_pcn1="prod sp 192.168.1.211 192.168.1.212 24 cxnode1 192.168.1.201"

(the ones with * are the variables I had to update when I created the cxnode2 template) 

Installing Solaris Cluster on the nodes

As the nodes were running in 32bit mode, (VirtualBox 1.7 should have 64bit support, which means this part won't be necessary in the future) I needed to use an internally available set of 32 bit kernel drivers in a packages called SUNWscka. Basically, install sc3.2 as usual using the installer, but don't configure anything, and don't configure a quorum device. Then pkgadd SUNWscka on both nodes, and reboot them both.

The rest

After that, I just followed Gordon's instructions for setting up the Quorum server, setting up the iSCSI devices on the hosts, mounting them on the cluster nodes, and creating a zfs pool using them.

Sadly, this now means I can play with and configure all sorts of things in a Sun Cluster environment from the comfort of my laptop!

tags: Solaris Jumpstart JET xvm virtualbox

There's been a vibrant discussion on the sysadmin-discuss(at)opensolaris.org alias about boot disk layouts, and it was suggested that we try to come to some form of consensus and document it.

Sun has a bunch of installation standards called EIS (Enterprise Installation Standards), and one of those standards is the EIS Bootdisk Standard. As one of the contributors to that document, I found it quite interesting to see that the sysadmin discussion was coming to similar conclusions about how a boot disk should be laid out as we had come to internally.

In the interests of sharing and getting to a common standard, I've now posted the relevant bits of the EIS standard on the BigAdmin wiki, and encouraged the members of the sysadmin-discuss alias to comment and contribute.

If you're interested, you should have a look at it, and join the alias to participate in the discussion.

tags: Solaris Bigadmin

 (I've reposted so that technorati picks up the tag we'll be using for this event)

(Hint, tag using suncec2007 and we'll be able to agregate all content using said tag)

Its that time of year again. CEC2007 will be in Las Vegas this year. I've never been, and I'm half looking forward to it, but as I think I have an addictive character, I'm a little scared of of Vegas.

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables. 

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

The theme this year is "Shift Our Universe, Our World, Your Move", sub themes being: Shift to red: (all the groovy bleeding edge stuff that GregP talks about, Shift to green: (all the nice eco-friendly stuff we've been doing), and Shift to grow: (which focusses on all the things we need to do to enable our growth (and our customer's as well))

We're also going to be giving out a bunch of prizes for participants and attendees of the conference. They'll be mainly based on level and quality of participation in the sessions. Stay tuned for more details.

As per last year, I'll be blogging my perspectives of the conference again. 

tags: Sun suncec2007

As per a previous article, I upgraded by home server to Nevada build 69, because I was about to make some changes, and wanted to be fully up to date before I did so.

I wanted to play with Confluence which is the wiki that Sun has chosen to run their wiki site. Confluence can run in a number of ways and connect to a number of different databases. Since I hadn't yet played with Postgres, I decided I would use that, and I also decided to run it using Tomcat instead of as a standalone app.

Given that I'm a bit of a neophyte when it comes to talking native SQL, I decided that I needed to run a tool of some sort to configure Postgres. I've used phpMyAdmin for MySQL for a while now, and figured there MUST be an equivalent for Postgres. Using my amazing powers of deduction, I googled phppgadmin, and lo and behold there it was: phpPgAdmin. All I needed was a php compatible webserver. Unfortunately the supplied apache2 with  Nevada isn't php ready, so I had to then decide.. do I compile myself, or do I find something easier. I vaguely remember hearing something about coolstack, which is "a collection of some of the most commonly used open source applications optimized for the Sun Solaris OS platform."

CoolStack 1.1 is available for both SPARC and x64, and contains pretty much all I needed, including tomcat. So a few downloads and pkgadds later, and I've got apache running, which means I can get phpPgAdmin running, which means I can perform my initial Postgres configuration, which means I can finally install Confluence under Tomcat, and everything's perfect. (Well it wasn't perfectly perfect, because I needed to read a couple of READMEs along the way, and a bit of on-line documentation, but it was fine.)

So, I've got tomcat listening on port 8081 as per normal and confluence sitting inside there as an applet talking happily to Postgres, and so I decide that I actually want to do the apache proxy thing so that I can have my confluence wiki appear to sit on my normal webserver port. This is quite easy to configure, you just need a few lines like this:

LoadModule proxy_module libexec/mod_proxy.so
LoadModule proxy_connect_module libexec/mod_proxy_connect.so
LoadModule proxy_ftp_module libexec/mod_proxy_ftp.so
LoadModule proxy_http_module libexec/mod_proxy_http.so
LoadModule proxy_ajp_module libexec/mod_proxy_ajp.so
LoadModule proxy_balancer_module libexec/mod_proxy_balancer.so
ProxyPass               /wiki           http://localhost:8081/wiki
ProxyPassReverse        /wiki           http://localhost:8081/wiki

(I don't actually know how many of the LoadModules I really need btw, I suspect I don't need the last 2)

Here comes the bad news. The apache2 that comes with CoolStack doesn't come with the mod_proxy.so modules, which means that I can't use it for proxying. I tried copying the  original mod_proxy.so modules to the coolstack one and loading it up, but when I tried to access the wiki, my system pretty much froze and the disks churned like mad. Had to kill the webserver to get things back to normal.

How did I fix? I reverted to the original Nevada supplied webserver, and configured the CoolStack based one to run on a different port which I can use when I want to use phpPgAdmin. At some point I'm going to have to solve the problem permanently, probably some further investigation, and trying to figure out exactly what's going wrong. But for now, its doing what I need, I just hope I don't want to run a php based app anytime soon. 

tags: Sun Solaris CoolStack AMP

 Its that time of year again. CEC2007 will be in Las Vegas this year. I've never been, and I'm half looking forward to it, but as I think I have an addictive character, I'm a little scared of of Vegas.

At the minimum, I'm going to leave my credit cards in the hotel room when I go out, so that I don't end up losing more than just my shirt at the poker tables. 

More importantly, I think the conference is going to be great. I've seen the list of some of the topics that will be presented.

The theme this year is "Shift Our Universe, Our World, Your Move", sub themes being: Shift to red: (all the groovy bleeding edge stuff that GregP talks about, Shift to green: (all the nice eco-friendly stuff we've been doing), and Shift to grow: (which focusses on all the things we need to do to enable our growth (and our customer's as well))

As per last year, I'll be blogging my perspectives of the conference again. 

tags: Sun suncec2007

Sun has just launched http://wikis.sun.com, which is great because it allows me to use it for putting all the JET info there rather than here.

I have created a space there called JET, which will become the primary Sun location for all information relating to the Jumpstart Enterprise Toolkit.

Have a look: Jet wiki

tags: Solaris Jumpstart JET