Thursday July 03, 2008

xVM Ops Center in a Virtual world...

Yet another little homespun project that will make people ask "Why?!". I need to get up to speed on xVM Ops Center to stay ahead of my customers. It also seems like a really cool tool for managing the dozen or so machines around my house. Yes, I have three kids, 9, 4, and 2, and one computer literate wife. Yet there are about a dozen machines up and running at any given time, plus a pile of virtual machines serving a variety of purposes. There are SunRays around the place, great gadgets for ordering car parts from the garage if you don't want to keep running inside and washing your hands to use a computer. Those spill-proof keyboards like the ones they use at Jiffy Lube rock for greasy fingers searching for 30 year old wiper switches on eBay.

Here is a little screenshot from within a xVM VirtualBox Solaris 10 VM on my desktop... Let your mind wander to the possibilities.

I'll bet you thought of xVM Ops Center as a potentially fabulous tool for managing all of your servers and possibly even virtual machines within those servers. But why not run the Ops Center itself from within a virtual machine? Hmmm... Think on that one for a while, and I'll be back with more details.

bill.

VirtualBox with a promiscuous floozy...

Everything was working just fine on my laptop machine, my VirtualBox VMs could see each other, Jumpstart and JET were just fine, and all of the VMs could see the outside world. When I moved my configs into my ridiculously over-configured desktop machine (see blahg entries from about 18 months ago), networking stopped functioning. Weird. In fact, my Windows host machine stopped communicating to the outside world as well. Ouch.

VirtualBox uses a "TAP" interface driver to bridge or NAT the VM networks through the WIndows host. On the Windows side, in order for this "TAP" adapter to talk to the outside world, it must use the "Network Bridge" or "MAC Miniport Adapter". If you select the TAP adapter, CTRL-click your network adapter that talks to the outside world, right mouse click and select "Add to Bridge", the two adapters are linked together into a single logical interface. Packets from one side can magically appear on the other side. In fact, you should be able to "snoop" from a Solaris virtual machine and see packets flying around on your external network. The final Windows XP Network Connections window should look something like this:

Yeah, that's how it is supposed to work. Mine didn't. I checked the Firewall settings, and even disabled it temporarily to make sure that it wasn't the problem. I Googled (and yes, that is a verb), and found nothing apropos. After several hours of head scratching, reading blogs, reading through the Microsoft technical docs, and banging my head against the keyboard, I stumbled on the answer. My adapters were not in "promiscuous mode". Promiscuous mode allows adapters to pass through all of the packets that they see, rather than just the packets that are addressed to this host. That sounds like what I need.

The magic mojo to make my bridged adapters go into promiscuous mode was fairly painless. Click on [Start] -> [Run] and type "cmd" to get a shell window on the XP host. In the shell window, I was able to run the netsh commands necessary to make the adapters go promiscuous, and to check that the changes were in place. This changes registry entries, so you should be able to just jump through these hoops once, and the settings should remain through reboots:

Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. E:\Documents and Settings\Bill Walker>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- ---------------------------------------------------------------------- E:\>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- 1 VirtualBox Host Interface 1 unknown 2 Local Area Connection 2 unknown ---------------------------------------------------------------------- E:\>netsh bridge set adapter 1 forcecompatmode=enable E:\>netsh bridge set adapter 2 forcecompatmode=enable E:\>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- 1 VirtualBox Host Interface 1 enabled 2 Local Area Connection 2 enabled ----------------------------------------------------------------------

Voila! My Windows network is back to a sane mode. My VPN connections to the office are alive and well. My VirtualBox VMs can now see the outside world. I now have my Frankenstein super-desktop (TBs of disk, 8G of memory, 4x Opteron cores) back in a happy state. As always, your mileage may vary, objects in mirror may be closer than they appear, no warranty expressed or implied, but I hope this helps someone save hours of head banging on fragile keyboards at some point.

bill.

Sunday June 29, 2008

VirtualBox, the VM JET Client Chronicles...

Here is the JET template for my simple, sample virtual machine example. Wow, I've been watching way too much toddler TV. I'm starting to rhyme things for no apparent reason.

# Ethernet can be obtained from the 'banner' command at OBP # # OS is one of the values you used to register the solaris media using # the add_solaris_location command # base_config_ClientArch=i86pc base_config_ClientEther=08:00:27:8C:D9:E0 base_config_ClientOS=S10u5

This gives us the ethernet address, chip architecture, and OS version to be used to install the client VM.

# # Client allocation # # The mechanism used to build this client; by default, the options listed # in /opt/jet/etc/jumpstart.conf will be tried; you should only set this # if this particular client needs to do something different. # JET supports bootp, dhcp, and grub as allocation options. # Currently grub is only supported on i86pc architectures. # base_config_client_allocation="dhcp"

Using "dhcp" allows me to do the PXE boot magic inside the VM intnet network.

############ # # JumpStart sysidcfg information # base_config_sysidcfg_nameservice=NONE base_config_sysidcfg_network_interface=PRIMARY base_config_sysidcfg_ip_address=192.168.1.202 base_config_sysidcfg_netmask=255.255.255.0 base_config_sysidcfg_root_password="JGnRL6GHcobuc" base_config_sysidcfg_system_locale="C" base_config_sysidcfg_timeserver=localhost base_config_sysidcfg_timezone="US/Eastern" base_config_sysidcfg_terminal=vt100 base_config_sysidcfg_security_policy=NONE base_config_sysidcfg_protocol_ipv6=no base_config_sysidcfg_default_route=192.168.1.1

This is the standard sysidcfg stuff. Same here as in Jumpstart, or bringing up a sys-unconfig'd machine.

# base_config_x86_nowin="yes" base_config_x86_console="" base_config_x86_disable_acpi="" base_config_x86_safetoreboot="yes" base_config_x86_disable_kdmconfig="yes" base_config_x86_confflags="" base_config_sysidcfg_x86_kdmfile=""

Some simple options for booting and setup. The comments and documentation explain all of these fairly well.

base_config_profile_cluster=SUNWCall base_config_profile_usedisk=rootdisk. base_config_profile_dontuse="" base_config_profile_root=free base_config_profile_swap=256

This is the "what to install" and the simple root disk configuration. I am reserving 256M for a swap partition, and putting the rest of the available space on the root partition.

That's the basics. You can do alot more with Template files, including the SVM and JASS modules. Read the comments in the make_template generated templates, read the docs, read the blogs, and do lots of copying and pasting.

There we go. We have now created a totally hands-off installation of a system inside the VirtualBox. If we configured the boot devices correctly (disk first, network second), then there was no interaction necessary between clicking the "Start" button on the VirtualBox GUI, and getting a login screen on the fully configured and running client system. Very cool stuff.

bill.

Saturday June 28, 2008

VirtualBox meets JET...

So first, the why. I was working on a project (described in detail in earlier blahg entries) that used JET/Jumpstart to maintain "golden images" of systems for provisioning in the application and web tiers. First reason for doing this was just to have a JET configuration that I could play with in my hotel room while waiting for room service. Sitting around the office behind the firewalls just to get access to the boxes can be a bummer after about 8pm.

Second reason for this, I'm a geek. Once I had the idea in my head, I just had to get it to work. Third reason, it is a really handy way to set up machine variances within VMs on your desktop or laptop for demos and testing software. Yeah, you can "clone" VMs, and copy the disk images and configs around. But you never know when you are going to "fat finger" a VM, or forget some important step when munging things by hand. Using a JET server allows you to save a config, re-use it, and always start with a fresh "machine" to work on, in a known state. You can even save off your favorite VMs as "flar" images to restore on demand, using compressed images can save *tons* of space, especially important when you are on a disk-limited road warrior laptop system.

Step one, download Sun xVM VirtualBox and read all of the assorted docs and technical info. Join the community, read the Wikis, read some blogs.

Step two, install the downloaded package on your system of choice. In my case, I am running it on my laptop and desktop machines, both under Windows XP. There are lots of folks running other host systems, read the info, read the blogs.

Step three, download your Solaris or OpenSolaris DVD image of choice. No need to burn the image on to DVD media, we'll just mount the ISO DVD image as a virtual DVD later. While you are cruising OpenSolaris.org, join some discussions, cruise through the forums, read some documentation, join the movement. (Can you tell I really like OpenSolaris?)

Step four, (well, maybe after you play around with VirtualBox a bit and get comfortable with it) create your Jet Server virtual machine (VM). Double click on the desktop icon, and then click the "New" button. For my configuration, I went with an 32G root disk, added in 32G for /export (only allocated from your hard drive as you use the space), 512Meg of RAM (I have 8Gig on my desktop and 4Gig in my laptop), and defined the CD drive to be a virtual drive, mounting the ISO image that we just downloaded.

The networks here are the tricky part. I am using a "Back End Network" (BEN) for management. I will keep my JET/Jumpstart traffic on the BEN, and use a second interface as a "public network". My BEN network is defined as "internal network" or "intnet" (inter-VM traffic only). My external interface could be NAT or host bridge, either way, the Windows host can get to the VM, and the VM can see the outside world (Firefox can get to Google). I configure the VM through the GUI, but then ran into a little "issue".

There was a bug in the earlier (1.3.X) version of VirtualBox that broke "intnet" network connections. A couple packets would fly back and forth between VMs, and then the connections would just disappear. You could snoop and see packets going out, and sometimes being received, but answering packets never made it back to the originating host. I don't know if that is fixed in the current version (1.6), but there is a workaround. In the Windows host, click [Start] -> [Run], and then type in "cmd" to get a shell window. Use "cd" to go to the VirtualBox installation directory (C:\Program Files\Sun\xVM VirtualBox in my case), and use the command line utility VBoxManage.exe. The magic mojo here is (for my VM named JETserver):

  "C:\...xVM VirtualBox> VboxManage modifyvm JETserver -nic1 intnet

Note that the network interfaces are "nic1" (my BEN) and "nic2" (my public interface).

Now just click the "Start" button, followed quickly by pressing "F12" on your keyboard to choose the boot device. This option only appears during the splash screen of the virtual BIOS, so you have a second or two to hit the key. Choose CDROM for your boot device and away you go installing Solaris as usual. You can also juggle the boot devices under [General] -> [Advanced] -> [boot devices] in the configuration pane. Make sure that if you created a second drive for the JET stuff (as I did) that you set it up during the install, or fdisk/format/newfs it when the machine is booted. Make sure that you set your boot devices again, or unmount the virtual CDROM device, or hit F12 really quickly again to boot from disk after the install.

There we go, a Solaris VM. Yay! Now we need to make it into a JET server.

Step five (OK, so four was rather long), download the Jumpstart Enterprise Toolkit (JET) software from the JET home on BigAdmin. BigAdmin is your friend. Cruise around, read the JET Wiki, read docs, join communities, etc. as usual. Also take a cruise through Mike Ramchand's blog, and check out the Yahoo! JETJumpStart group.

Step six, follow the instructions for installing JET, using that ISO file again as a virtual CDROM device. Make sure you run:

/opt/SUNWjet/Products/base_config/solaris/make_dhcp 

to get the DHCP and PXE boot stuff in place, since we will be using those goodies to boot our client VMs.

The documentation for JET is included in the package in PDF format. Read it (at least give it a cursory pass through) before continuing. The simple stuff is really simple. The more complex and intricate operations can get tricky.

Step seven, create a new VM to be a JET client. This is simple, here is a screenshot of my JSclient machine to show the settings (don't forget to jump out to the command line again for VBoxManage.exe after you configure it through the GUI):

Take note of the ethernet address of your virtual adapter, and use that in your JET template. Make sure you assign the IP address of your JET client in the template file to be on the same network as your BEN "intnet" connection on the JET server VM. If you want to get fancy, you can add a public interface to the JET client configuration and add that to the JET template as well (DHCP or fixed IP with default router and DNS server info in the template).

Step eight, boot the sucker, hit "F12", and choose network boot (PXE) (or again, juggle your boot devices under the [General] -> [Advanced] -> [boot devices] configuration pane). If things don't work as planned, you can snoop your nic1 from the JETserver VM to debug things. That's it. Simple. I'll post some more screen shots and details of my template files as time permits.

bill.

Friday June 27, 2008

It's a Digital World (and then some!)...

Kudos to my friend Alok Rishi for presenting us with this evidence of blue collar digital overload from Hong Kong.

Wow. I especially like the pink one with the tassels. I wonder if that is the new quad-band Barbie phone?

bill.

Thursday June 26, 2008

TiVo detour...

So today we take a slight detour from the JET and Sun related geeky blahgs. Tuesday night my Series 1 DirecTivo decided to go Kablamo (with all due credit to Steve Matchett and David Hobbs). Wednesday morning, it was doing a "powering up, please wait" and GSOD (Green Screen of Death) loop. The GSOD was lasting about 3-4 seconds, so I knew something was terribly wrong.

This is a hacked DirecTivo, with one 30G (original), and one 160G PATA drive. I hacked the beast way back when 160G drives were the biggest and baddest. Remember those days? Back when vinyl records were just hitting 33 1/3 RPM, and the Edsel in your driveway was all the talk of the neighborhood? Well, maybe not that long ago, but at least 8-9 years. Not bad, a cheap "desktop quality" drive lasting 9 years of continuous and hard use. I guess I got my money's worth out of it.

So, do we go out and get a new unit, or try to fix the old one. Yep, I'm a geek. I need to fix the old one first, and then I might decide to go for the new unit. Step one, take out the drives, hook them up to a PC and see what's happening. Fortunately, they were both Maxtor drives, so I was able to use Seatools to do some diagnostics. The Windows version found major damage, so I grabbed the DOS version. The DOS version can fix bad blocks for you. Nice.

Yep. Hundreds of bad blocks in the middle of the drive, but only one bad sector around 66 or so, likely in the middle of the OS or at least the TiVo's system software. Bummer. I couldn't find the TiVo tools that I had last time, but I did remember that an old Sun employee is running a TiVo hack/upgrade/fix-it business now. I hit his website at DVRupgrade (Tell Lou Jacobs that Bill sent you his way if you find this helpful) and grabbed a copy of "InstantCake" just in case I couldn't fix the box, or pull a decent backup off of it. InstantCake will install a real, licensed TiVo OS image on to generic store-bought hard drives (of any size and lineage) for you. This is a nice, legal way to upgrade an old 30 or 80 hour Tivo up to hundreds of hours of standard resolution recording.

So now I'm covered. If I *totally* screw things up, I know I can go back to a "factory fresh" installation. Step 2, download the Tivo MFS Tools CD. There are good instructions and links to the software here. Use Google to find other sources of info and software, and as always, your mileage may vary. The MFS Tools (now open sourced from what I am reading) come on a bootable Linux Live-CD. Unplug your hard drives, plug in your Tivo Drives on the IDE cables, and boot from the CD. Leave a FAT32 drive (or grab an old one off of the shelf) to write backup files to if you so desire. Nice stuff.

So I stick on a 20G piece of junk WD Caviar out of a 1998 vintage Windows machine (yeah, it is time to clean out my office), and my Tivo drives. In fact, to keep my desktop machine safe and continue doing real work while things run, I grab a vintage 1999 P-III 933 Mhz Dell Optiplex 100 machine out of the garage to do the rest of this munging with. I know which drive is bad thanks to Seatools (the first drive, or Master). I use mfsbackup to save off my OS and settings. One warning about a bad block, but it was apparently in the SuperBlocks or directory structure, because it continued successfully. Yay, that much is safe. Now I can try to save everything, including about 200 hours of Dora, Bob the Builder and Hannah Montana for my kids. Imagine at this point how incredibly *miserable* my kids are without any TiVo in the living room, stuck with only 4 other TVs and DVD players!! The horror...

About 5 hours of "dd conv=noerror,sync if=/dev/hda of=/dev/hdb", and we'll give it a shot. Yeah, hundreds of I/O errors, but what the heck. From my backup, I know that most of the bad blocks are in the data. I had replaced the 30G drive with an old 40G drive from the depths of my filing cabinet. Free is good, and 5400 RPM cool-running drives are hard to find these days. I ran "mfsadd -r 4 -x /dev/hdb /dev/hdc" to marry my drives back together and add in the extra 10G or so as usable space. Pop the drives out of the PC and put the TiVo back together. Serious sidebar. Make darn sure you plug the fan back in!! You only make that mistake once, but it will melt the DirecTV access card in the back of the machine if you don't.

Drives back in, power on... Good old "powering up" messages, and into a GSoD. I expected this, as I just skipped over bad blocks, and I know one was in the system software. Hopefully it will remap or recover and move on. About 10 seconds into the GSoD, the machine reset. "Powering up", and GSoD. Again, not unusual, as I am expecting it to do the TiVo equivilant of an fsck now to get some sanity back in the data areas where the crash really ate some disk. About another 15 minutes or so, and another reset, "powering up", and... drum roll please... Success. My TiVo is now up and running. I am sure that some of my programs are trashed, or contain holes that will cause blackouts or stutters, but at least my Dora, Bob the Builder, and Hannah Montana collections are back online.

If you found this blog via Google because your TiVo is Kablamo, I hope this helped. If you just read it for grins, then I hope your day was better than mine.

To keep things really cool, I even installed (well, cut and snipped) a "chimney" over top of the gap between the drives in the upper case cover, and glued on (Silicone caulk is your friend) a temperature sensitive, variable speed PC case fan wired into the drive power leads. Yeah, it is a bit weird looking, but I'll bet most folks didn't get 8+ years of use out of their TiVo drives (and definitely not when one of them is a 7200 RPM drive). I'll take some pics of that little hack one of these days and post them here as well.

bill.

Wednesday June 25, 2008

Virtualbox on the run...

More details later, but I wanted to scribble a quick note on this one. I was able to get Jumpstart and JET running within Virtualbox. Some might ask (and rightly so), "Why?".

I am a paying customer of VMware Workstation. I bought a personal license with my own hard earned dollars (euros, drachma, baht, etc.), and use the heck out of it on my desktop workstation at home. I tried Parallels, but ran into a couple "issues" running with a Windows host that annoyed me to the point that I removed it. After all, my VMware Workstation was running just fine. When Sun acquired the VirtualBox stuff (now xVM Virtualbox), I downloaded it and used it quite a bit. Big wins for me, much smaller install size (great when you are hanging off the end of a EVDO wireless broadband link), very active user and developer base, and the fact that it is a release based on an OpenSource project.

My VMware Workstation license has finally run out of free upgrades. I'm running on a 32bit XP machine, so VMware Server (the free license) won't work for me. Throw in the 450+MB download size, the Microsoft licensing costs to upgrade my road warrior laptop, and this really wasn't an option. Virtualbox, here we come.

In theory, I could have converted my existing VMware Workstation VMs to Virtualbox (good reviews and details here). I did my research, saw some good how-to instructions and was fully prepared to go that way. Turns out, for my stuff, it was just as easy to run a quick ufsdump and stick a backup of my "working space" out on my Windows XP host. I just did a ufsrestore to get my goodies back under my Virtualbox VMs. Another neat feature of this method, I can use "ufsrestore -i" to pick and choose what exactly I want restored (I tend to get a bit of clutter in my workspaces).

But wait. You work for Sun. Why aren't you running Solaris native on your laptop? Yeah, I hear that alot. I was one of the hardcore Solaris-only laptop road warriors going back to Solaris 2.5. When Sun decided to un-fund laptop related Solaris work, a bunch of us continued in our spare time to keep things working and even slightly improving for road warriors. When Sun decided to do away with x86 support (only for a short time before reversing the decision with a vengeance), we kept the builds going, and used our own machines for testing.

Truth is, I'm old. I don't have the time and energy to muck around with my desktop/laptop these days. Windows (and probably Macs) just work when you plug them into hotel ethernets. They just work with cellular based broadband. Customers generally have a tools CD sitting around with their VPN software and configs ready to install under Windows. So my "primary hypervisor" is Windows XP. It works great for plugging me in, setting up hardware and networks, (playing my favorite Windows based games) and then running Solaris under Virtualbox so I can get real work done.

Next entry, taming the beast... getting JET to work under VirtualBox (and why).

bill.

Tuesday June 24, 2008

JET handoff to N1SPS / N1SM / DI...

The system side of things looks fairly good at this point. We have created our pile of packages to install, patched everything, decided on a disk configuration, tested it, and bundled the installation scripts into the system image "flar" that the N1SPS/N1SM folks are going to use in provisioning our target servers. Not bad.

Now is the time that we expect little problems and conflicts to creep in. The first one is expected, and (fortunately) we had a little time to play around to find out what would and wouldn't work. Simple problem, we needed to create and mount all of those filesystems as part of the installation of the global zone, but didn't know what the zone names would be. Since tradition puts the "zonepath" at /zones/[zonename], we will need to do some juggling. It turns out that N1SPS and N1SM think that, by default, zones will have a zonepath that is /zones/[zonename], our choices are to either re-write a bunch of tested local, supported, and standard SPS/SM goodies to deal with this dilemma, or fix it on the system side before SPS/SM come into play.

We first tried using "ln" and "ln -s" style links. These broke things in very interesting ways (as suspected, but it was worth a try). The zonecfg/zoneadm utilities were unable to figure out where the disk space was coming from, and how much of it there was to use. Oops. My next brainstorm was to use "lofs". The lofs "loopback filesystem" allows you to do lots of tricky things like mount ISO images of CDs and DVDs. It also allows you to "mount" directory trees of stuff from one place onto another "mount point". This functionality is leveraged by zones themselves to create the filesystems in your zone config (especially noticeable with full root and multi-filesystem zones).

Pretty simple solution, and rather elegant. As a pre-install task for any zone (let's call ours "myzone"), we mount the zone disk space that we created earlier (let's use ZONE123 for this example):

myhost# ./zonelink myzone ZONE123 Mounting /zones/ZONE123/ROOT on /zones/myzone myhost#

And here is the quick and dirty zonelink script. In retrospect, I would have mounted the soft partitions someplace other than /zones. This would have made good separation of "mountpoints" from a global zone perspective and "zonepaths" and zone filesystems. Something like /zonespace would have made life neater and cleaner in the long run. Live and learn.

#!/bin/sh # # zonelink # 1.0 - Bill Walker < bill.walker@sun.com> # # Use lofs (7fs) to mount the zone space on the zonepath # ########################################################### # BN=`/usr/bin/basename ${0}` # # Check Args, do Usage... # if [ $# -ne 2 ] then echo "${BN} : Usage : " echo " ${BN} [ZONENAME] [ZONENUMBER]" exit 111 else # Collect args into vars # ZONENAME=${1} ZONENUMBER=${2} # # Make sure that the mountpoint exists... # if [ ! -d /zones/${ZONENAME} ] then echo "Making mountpoint /zones/${ZONENAME}" mkdir /zones/${ZONENAME} fi # # Make sure the source dir exists... # if [ -d /zones/${ZONENUMBER}/ROOT ] then # # Use lofs to mount zone space on /zones/[zonename] # echo "Mounting /zones/${ZONENUMBER}/ROOT on /zones/${ZONENAME}" /usr/sbin/mount -F lofs /zones/${ZONENUMBER}/ROOT /zones/${ZONENAME} else echo "No such source ROOT /zones/${ZONENUMBER}/ROOT ..." exit 112 fi fi

I could get alot fancier with this and add a bunch more error checking. I really should check to see if the zone space is already being used before I try to lofs it to the zonepath. Something like:

# # Count the number of times that the ZONENUMBER/ROOT is in # the output of /usr/sbin/mount # MNTLINES=`/usr/sbin/mount | grep "/zones/${ZONENUMBER}/ROOT" | wc -l` # # Make sure that there is only one entry for ZONENUMBER/ROOT # in the output of /usr/bin/mount, or error and exit # if [ ${MNTLINES} -gt 1 ] then echo "${BN} : /zones/${ZONENUMBER}/ROOT already mounted somewhere?" exit 113 fi if [ ${MNTLINES} -lt 1 ] then echo "${BN} : /zones/${ZONENUMBER}/ROOT not a mounted FS?" exit 114 fi

OK, that snippet is off the top of my head and totally untested, but I am using it as an example of the myriad of things that could go awry in this part of the process. There are a ton of dependencies, but since we own the platform, and we presume that it is under our control (no one has been mucking with it), and it is more than likely only a few minutes or hours since a fresh installation... We can probably take a few relatively safe shortcuts at this point.

The rest of the filesystems in our zones will be configured using zonecfg, and just work. The root filesystem was the only one that caused weirdness. Once the root filesystem is mounted in the right place, zonecfg takes care of all of the rest of the lofs filesystem mounts for us and "just works".

bill.

Cleaning house (and ssh known_hosts)...

One issue that nagged us for quite a while during this project was cleaning up the provisioning servers (JET/Jumpstart or N1SPS/N1SM). Since we were developing in a lab environment, our provisioned and JET'd hosts were re-installed hundreds of times over the course of the project. As part of the installation of a target host and integrating it into the management framework, several user accounts use ssh to pop into the machine and twiddle bits (add packages, change configuration files, move stuff around, start services, etc.). I thought I would add this blahg entry so that others might Google the symptoms and find some relief.

Have I mentioned that Google is my favorite debugging tool? If you are having issues, odds are that you are not the first to stumble upon them. Even if it is a "silly user error", someone has probably asked about that exact (or a very similar) error message or symptom in some newsgroup, support alias, or documented it in an FAQ somewhere. Even if the answer is buried somewhere on docs.sun.com, Google will likely find the answers for you.

The symptom that we were seeing is rather simple. When a host is installed (or re-installed), ssh-keygen is run on first boot to establish the credentials needed for SSH to work. Since this machine is re-using an IP address / hostname that has already been seen as a "real machine", remote machines trying to connect to it will already have an entry in ~/.ssh/known_hosts associated with this IP address. This causes an interesting error message when you try to SSH into the machine, informing you that there is a key mismatch, and that someone is possibly doing something very evil and spoofing the target machine. Connection closed.

Simple answer, when you are going to re-install or re-provision a machine, just "cd" all over the place and delete all of the known_hosts entries for that IP address. Yeah, like we are going to remember to do that, or remember all of the accounts that have used SSH to access that machine. With 7-8 re-installs happening every day in our environment, this was ugly. We had failures 2-3 hours into the provisioning testing with N1SPS/N1SM because application stuff would fail to install/configure with SSH errors. Lots of wasted time.

So here is my simple answer. A stupid little shell script that is run on the JET / Jumpstart server and/or the N1SPS/N1SM server in the lab when re-using a target system and IP. Yes, you can put this into JET to run when a "make_client -f" is run. Yes, you can have N1SPS/N1SM run this automagically as part of preparing to provision a new host. Yes, this is a really simple task and a really simple shell script, but if it saves someone 2-3 hours on failed provisioning testing, then I am happy to waste bandwidth to stick it on my blahg.

#!/bin/sh # # # 1.0 - Bill Walker # Clean out ~/.ssh/known_hosts for re-provisioning a # machine that already has SSH host keys installed # for a variety of user names. # ######################################################### # # Save off command name in $0 # BN=`/usr/bin/basename ${0}` # # Users to clean out... # USERS="root n1sps" # # Check arguments # if [ $# -ne 1 ] then echo ; echo "Usage: ${BN} [hostname|IP address]" ; echo exit 1 fi # ######################################################## # HOST=${1} # ######################################################## # # DELETE this if you don't care... # Pause and display... # echo "About to remove known_hosts entry for host ${HOST}" echo " for users ${USERS}" echo; echo "< Return > to continue" read XXX # #END DELETE THIS IF YOU DON'T CARE##################### # # for ACCOUNT in ${USERS} do # # Using getent just in case the user is LDAP and # not /etc/passwd # HOMEDIR=`/usr/bin/getent passwd ${ACCOUNT} | cut -d: -f6` # # If the home dir exists... # if [ X${HOMEDIR} = "X" ] then echo "${BN}: ${ACCOUNT} doesn\'t exist (continuing)" else # # If home dir and known hosts exist... # KH="${HOMEDIR}/.ssh/known_hosts" if [ -d ${HOMEDIR} -a -f ${KH} ] then if /usr/bin/grep "^${HOST} " ${KH} >/dev/null 2>&1 then # # Strip out host entry for ^HOST from KH # echo "${BN}: Processing ^${KH} for ${HOST}" /usr/bin/sed -e "/^${HOST} /d" ${KH} >/tmp/kh.$$ /usr/bin/cp ${KH} ${KH}.orig /usr/bin/mv /tmp/kh.$$ ${KH} else if /usr/bin/grep ",${HOST} " ${KH} >/dev/null 2>&1 then # # Strip out host entry for ,HOST from KH # echo "${BN}: Processing ,${KH} for ${HOST}" /usr/bin/sed -e "/,${HOST} /d" ${KH} >/tmp/kh.$$ /usr/bin/cp ${KH} ${KH}.orig /usr/bin/mv /tmp/kh.$$ ${KH} else echo "${BN}: No ${KH} entry for ${HOST}" fi fi else echo "${BN}: ${USER} doesn\'t have a ${KH} (continuing)" fi fi done

Just edit "USERS" to be the list of users that you want to clean out and away you go. This script cleans out entries that have the IP address at the beginning of the line, and entries that have "hostname,IP". It saves off a copy of the known_hosts file in ~/.ssh/known_hosts.orig just in case something goes awry. I could modify this to go through /etc/passwd or something like that to get the USERS list, but what do you want for a quick hack written at 11pm in a hotel room in the middle of the desert? As always, your mileage may vary, objects in mirror may be closer than they appear, and (of course), there is no warranty expressed or implied. Use at your own risk. I hope it saves someone some aggravation and time.

Oh yeah, any comments about how my shell scripts look like old school top-down FORTRAN are definitely >/dev/null 2>&1. I'm old, I write linear code, it (mostly) works.

bill.

Friday June 20, 2008

Pass the buck (or a message) using motd & EEPROM...

Another nifty challenge that we had on this project was juggling resources. We had about 8 machines in our development lab, and about 30 engineers working on them at any given time. One machine is reserved as the Jumpstart/JET server, one as the N1SPS/N1SM server, and two for cluster development. One machine was mostly dedicated to the "golden flar image" development, leaving three machines for everyone else to scramble for as target server system for deployment and functional/unit testing.

This qualifies as the hack of the day for me. We could all send emails around, but the volume and conflicting needs would be overwhelming. We could (and did) maintain a spreadsheet and whiteboard of who had what assets reserved for what periods of time. We could (and did) keep the assets in the project plan docs, though "issues" and "need a machine for debugging this new error that just popped up" often makes those things out of date before they can be sent to the printer.

We ended up assigning a "group owner" to the machines as they were allocated. So the "Solaris" team might own a machine for a couple hours or a couple days. That person was responsible for knowing who was working on the machine, the state of the machine, and what tasks were to be completed before giving the machine back to the pool for reassignment.

Simple things like adding/deleting/cloning zones, messing with ndd settings, playing with secondary network adapters to configure IPMP aren't conflicts. Re-deploying or JET installing a machine that people are counting on could set teams back several hours or days. How can you pass a message other than having people run down the hall, and pop into the work rooms one at a time screaming "who just rebooted my machine, what's happening?"

You can place a message in the /etc/motd file (yeah, it is old school, but it works) letting everyone know what is happening on the machine. The "group owner" of the machine at any given time is responsible for placing notes there to keep incoming users informed as to what activitied and instabilities might exist on the machine, as well the the contact info for the machine owner.

somehost# cat /etc/motd Machine: somehost IP: 10.68.75.9 Owner: Bill Walker (703.555.1234) Purpose: JET/flar work Current State: Al: Doing ndd settings testing Jim: Working on Packaging tools Bill: Working on Issues 118 and 181, SSH key mismatch issue Working on Issue 58, root homedir changes halfway through provisioning process.

The other nifty item is the EEPROM's oem-banner variable. If the /etc/motd can inform folks coming in as to what is being done and by whom, the EEPROM oem-banner can alert folks who might be coming in through the ILOM and trying to re-provision or re-JET the machine out from under you. The last thing you want after working on a machine for 3-4 hours is to see "Connection closed", and find out that someone just typed "boot net - install" at the ok> prompt.

Historically, the OBP's oem-banner variables were created so that hardware manufacturers could "relabel" machines and resell them. Replacing the skin of the machine and putting a new logo on it was easy, but the console power-on message at POST time had a banner identifying the machine model and manufacturer, and on graphic (frame buffer) equipped systems, a graphic image depending on the machine and frame buffer type. The oem-banner and oem-logo variables allow machines to display different banners and graphic logos, essentially hiding the manufacturer of the machine for OEMs. We are going to appropriate these data elements for our own use in this case.

somehost# eeprom oem-banner="STOP!!! System Group box for flar development somehost> call before re-installing! Bill Walker (703.555.1234)" somehost# eeprom oem-banner?=true somehost#

Now when someone goes to the console to re-install the box, they will see:

{0} ok {0} ok boot net - install STOP!!! System Group box for flar development call before re-installing! Bill Walker (703.555.1234) Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0 File and args: - install 1000 Mbps full duplex Link up Requesting Internet Address for 0:14:4f:d3:9a:0 Requesting Internet Address for 0:14:4f:d3:9a:0

Since the "Requesting Internet Address" stuff takes several minutes, they should have plenty of time to go back to the ILOM, issue a "break -y", and make the call before the disks get scribbled over.

Your mileage may vary, but I thought this was a good idea.

bill.

Thursday June 19, 2008

JET checkpoint, our milestone...

At this point, we have the packages and patches that we want, and the disk configuration that we want. We have a running system that we can fiddle with and install configuration files into. From this point forward, the system flar becomes evolutionary. Files like complex resolv.conf, NTP configurations (ntp.conf), passwd, shadow, netmasks, networks, sshd_config and the like can be configured on the JET provisioned host and then wrapped up into our "Golden Image" flar for N1SPS/N1SM to use.

For this project, we tried to maintain a line of separation between the "system" side of the configurations, and the "deployed applications" side. Within the deployed applications pile, we have two groups: IT and management applications, and end-user applications. Things like SunMC, backup tools, and administrative utilities are deployed through the IT and management steps. End-user applications (web servers, application servers, message queue) will be provisioned through other steps owned by different groups under N1SPS and N1SM (with DI Dynamic Infrastructure). Rather than use both JET and N1SPS/N1SM to maintain our system configuration, we concentrated on the JET side, and only worked within the deployment frameworks where it was necessary.

One accidental side effect that we leveraged early on was provided by the JET framework itself. All of the JET generated scripts that twiddle the bits after the Jumpstart of the system are installed on the JET installed machine. In /var/opt/sun/jet, we find the post_install scripts that were used to create the system. This includes the scripts to set up the root disk mirroring, the metadbs, and the zone partitioning. The scripts for other configuration tasks, generating ssh keys, setting ndd configurations, enabling/disabling services, installing our JASS/SST driver, adding services that will run a JASS audit every time the system boots, and our /etc/system setup script that I noted earlier, are also present.

Since all of this work was done to automate the installation and make our repetitive configuration tasks easier and safer, why not leverage that work in the deployment frameworks? So our flar not only contains the software that we want to be installed, and the configurations of the services within that system, but it also contains the necessary scripts to deploy that image and those configurations on a piece of hardware, and regenerate alot of the settings and configuration dynamically. Very cool.

As an example of how this is a timesaver for us, we had to juggle the zone soft partition sizes in mid-project. If we were using the N1SPS and N1SM (including DI Dynamic Infrastructure goodies) plans to deploy the disk configurations, we would have had to juggle configuration information for all of our configured hosts in the test environment. In our case, we modified the JET template with the new sizes, ran the "make_client -f", did a "boot net - install" on our test machine, and then rolled up a new flar revision for the deployment folks to use with the new disk configuration embedded in the flar through the inclusion of the JET generated scripts. Grand total time to do this was a couple hours, and because this was a "system side" change, the IT/management and end-user applications folks under N1SPS and N1SM were never impacted by the change. In fact, they didn't even notice that the change had taken place. Very cool.

Late in the project, we did repeat all of our evolutionary steps on a single flar revision just to test our release notes and documentation. We went back and installed the packages we wanted from the DVD, and applied all of our documented changes to produce the "final flar" image from scratch. This was a great way to test our documentation and our procedures, and did uncover a few issues for us. Still, grand total time to produce the final system image flar with embedded scripts was one day, including testing. Nice.

One of my co-workers (Hi Jim!) became deeply involved with the packaging and cluster mechanics of Solaris distributions in this project. He wrote several tools (that I will let him write about at a future date and hopefully contribute to OpenSolaris) for juggling and debugging the package dependencies and installation order information. He wrote one tool in particular that I found incredibly useful. The tool "cooks" the package information from the installation media, takes a snapshot of the pkginfo from an installed machine, and creates the ordered list of packages to add necessary to satisfy all of the pkgadd dependencies. This will allow us to create our own "metaclusters". So we can now add "SUNWCgolden" to the standard "SUNWCall, SUNWCprog, SUNWCuser, and SUNWCmin" Jumpstart options. No more "packages to add" and "packages to remove" as post-install tasks to generate our flar from the installation media. Jim rocks.

bill.

In the zone with SVM and JET...

We now have a decent OS image, our root disk layout finished and mirrored, along with the associated metadbs live. You DID test at this point and make sure that the template file checks out and the target machine installs, right?

The partitions that we need to use for zone allocations are created for us, slice s6 on each of our 8 disks. The root disk and the root mirror disk have about 80G in s6, and the remaining 6 disks have about 143G in s6. We need to glue all of that free disk space together into a giant mirrored metadevice. We'll call it d100. In our sds_metadevices_to_init variable, we need to add:

d101 d102 d100 d101:4:1:c1t0d0s6:1:c1t2d0s6:1:c1t4d0s6:1:c1t6d0s6 d102:4:1:c1t1d0s6:1:c1t3d0s6:1:c1t5d0s6:1:c1t7d0s6 d100:-m:d101

This created metadevices d101, d102 and d100. Metadevice d101 would be defined as a 4-way concatenation of partitions c1t0d0s6, c1t2d0s6, c1t4d0s6, and c1t6d0s6. Metadevice d102 would be defined as a 4-way concatenation of partitions c1t1d0s6, c1t3d0s6, c1t5d0s6, and c1t7d0s6. We didn't see a real need for striping in this configuration for performance, and we want to retain the option of stacking more disk partitions on top of the concatenation at some point, if that becomes an issue. Remember that these disks are on a single controller, with the throttles and bottlenecks associated with that hardware limitation.

The two metadeviced d101 and d102 were then mirrored into a new metadevice named d100. This could have been accomplished by the command lines:

# metainit d101 4 1 c1t0d0s6 1 c1t2d0s6 1 c1t4d0s6 1 c1t6d0s6 # metainit d102 4 1 c1t1d0s6:1 c1t3d0s6 1 c1t5d0s6 1 c1t7d0s6 # metainit d100 -m d101 # metattach d100 d102

Now that we have a ~500GB disk partition called d100, we need to slice out the soft partitions. Since we were unable to use ZFS, using SVM softpartitions gives us the maximum flexibility possible when we slice out the zone partitions. For now, we will be creating fairly standard sizes and mountpoints for the zones to be created on, but that might change later.

The first zone (100) is for the Message Queue, and will contain 4 filesystems (two, plus Live Upgrade space). Zones 101 through 135 will use the same 6 partitions and sizes. Naming for the zone space metadevices is d[zone number][partition number].

d1000 d1001 d1002 d1003 d1000:-p:d100:4500M d1001:-p:d100:4g d1002:-p:d100:4500M d1003:-p:d100:4g d1010 d1011 d1012 d1013 d1014 d1015 d1010:-p:d100:3200M d1011:-p:d100:1800M d1012:-p:d100:2g d1013:-p:d100:1g d1014:-p:d100:3200M d1015:-p:d100:1800M ... d1350 d1351 d1352 d1353 d1354 d1355 d1350:-p:d100:3200M d1351:-p:d100:1800M d1352:-p:d100:2g d1353:-p:d100:1g d1354:-p:d100:3200M d1355:-p:d100:1800M"

This is the equivilant of running over 200 "metainit d#### -p d100 [size]" commands by hand. I created this section with a quick and dirty shell script that did a loop, counting up each zone, and containing a loop with echo's that created the lines for that zone. Ugly, but quick and effective.

# Creating filesystems # # If you wish to newfs any UFS filesystems, then you can do them from here. # As a side effect, the devices need not be DiskSuite ones.... # # sds_newfs_devices: Space separated list of devices to newfs - full paths # # If you wish to specify extra newfs options, then use a custom script to # perform the newfs etc. THIS WILL ONLY CREATE UFS FILESYSTEMS! # sds_newfs_devices=" /dev/md/dsk/d1000 /dev/md/dsk/d1001 /dev/md/dsk/d1002 /dev/md/dsk/d1003 /dev/md/dsk/d1010 /dev/md/dsk/d1011 /dev/md/dsk/d1012 /dev/md/dsk/d1013 ... /dev/md/dsk/d1350 /dev/md/dsk/d1351 /dev/md/dsk/d1352 /dev/md/dsk/d1353 /dev/md/dsk/d1354 /dev/md/dsk/d1355"

This tells JET to newfs all of those metadevices that we just softpartitioned out of the d100 space. This is the most time-consuming piece of the machine build, lasting about 10-15 minutes. Again, this section was initially created with a quick and dirty shell script full of "for" loops and "echo" statements.

# # sds_mount_devices: space separated list of tuples that will be used # to populate the /etc/vfstab file # # tuples of the form # # blockdevice:mountpoint:fsckpass:mountatboot:options # blockdevice: /dev/dsk/.... or /dev/md/dsk/.... # mountpoint: /export/big_raid_device # fsckpass: 5th column from /etc/vfstab # mountatboot: "yes" or "no" # options: 7th column from /etc/vfstab # sds_mount_devices=" /dev/md/dsk/d1000:/zones/ZONE100/ROOT:2:yes:- /dev/md/dsk/d1001:/zones/ZONE100/var:2:yes:- /dev/md/dsk/d1002:/zones/ZONE100/ROOT_LU:2:yes:- /dev/md/dsk/d1003:/zones/ZONE100/VAR_LU:2:yes:- /dev/md/dsk/d1010:/zones/ZONE101/ROOT:2:yes:- /dev/md/dsk/d1011:/zones/ZONE101/var:2:yes:- /dev/md/dsk/d1012:/zones/ZONE101/logs:2:yes:- /dev/md/dsk/d1013:/zones/ZONE101/apps:2:yes:- /dev/md/dsk/d1014:/zones/ZONE101/ROOT_LU:2:yes:- /dev/md/dsk/d1015:/zones/ZONE101/VAR_LU:2:yes:- ... /dev/md/dsk/d1350:/zones/ZONE135/ROOT:2:yes:- /dev/md/dsk/d1351:/zones/ZONE135/var:2:yes:- /dev/md/dsk/d1352:/zones/ZONE135/logs:2:yes:- /dev/md/dsk/d1353:/zones/ZONE135/apps:2:yes:- /dev/md/dsk/d1354:/zones/ZONE135/ROOT_LU:2:yes:- /dev/md/dsk/d1355:/zones/ZONE135/VAR_LU:2:yes:-"

This section creates the /etc/vfstab entries for the partitions that we just created. Yes, we are mounting the Live Upgrade space for both the global zone, and for our full root local zones, but that keeps folks from stealing space from my d100 or mounting those allocated partitions for other nefarious uses. The space is there and mounted into the global zone for when I need to use it for LU. Philosophical arguments on this item > /dev/null and as always, your mileage may vary.

Now we should be ready to cook our JET template and give it a shot:

jetserver# pwd /opt/SUNWjet/Templates jetserver# ../bin/make_client -f myhost Gathering network information.. Client: 10.68.75.9 (10.68.75.0/255.255.255.0) Server: 10.68.75.23 (10.68.75.0/255.255.255.0, SunOS) Solaris: client_prevalidate Clean up /etc/ethers Solaris: client_build Creating sysidcfg Creating profile Adding base_config specifics to client configuration Adding flash specifics to client configuration FLASH: Modifying client profile for flash install FLASH: Removing package/cluster/usedisk entries from profile Adding eiscd specifics to client configuration EISCD: EISCD: No profile changes needed Adding sds specifics to client configuration SDS: Configuring preferred metadevice numbers Solaris: Configuring JumpStart boot for myhost Starting SMF services for JumpStart Solaris: Configure bootparams build cleaning up preexisting install client "myhost" removing myhost from bootparams updating /etc/bootparams Force bootparams terminal type -Restart bootparamd Running '/opt/SUNWjet/bin/check_client myhost' Client: 10.68.75.9 (10.68.75.0/255.255.255.0) Server: 10.68.75.23 (10.68.75.0/255.255.255.0, SunOS) Checking product base_config/solaris Checking product flash FLASH: Checking nfs://10.68.75.23/export/flars/SPARC-App-Tier-20.flar Checking product custom Checking product eiscd Checking product sds -------------------------------------------------------------- Check of client myhost -> Passed.... jetserver#

YAY!!! We now have a template that (according to JET) might actually work. Let's go over to the target machine and give it a shot:

sc> break -y sc> console Enter #. to return to ALOM. {0} ok {0} ok {0} ok boot net - install  Boot device: /pci@0/pci@0/pci@1/pci@0/pci@2/network@0 File and args: - install 1000 Mbps full duplex Link up Requesting Internet Address for 0:14:4f:d3:9a:0 Requesting Internet Address for 0:14:4f:d3:9a:0 Requesting Internet Address for 0:14:4f:d3:9a:0 ... blah blah blah ... Using sysid configuration file 10.68.75.23:/opt/SUNWjet/Clients/myhost/sysidcfg Search complete. Discovering additional network configuration... Completing system identification... Starting remote procedure call (RPC) services: done. System identification complete. Starting Solaris installation program... Searching for JumpStart directory... Using rules.ok from 10.68.75.23:/opt/SUNWjet. Checking rules.ok file... Using begin script: Utils/begin Using derived profile: Utils/begin Using finish script: Utils/finish Executing JumpStart preinstall phase... Executing begin script "Utils/begin"... Installation of myhost at 09:45 on 19-Jun-2008 Loading JumpStart Server variables Loading JumpStart Server variables Loading Client configuration file Loading Client configuration file FLASH: Running flash begin script.... FLASH: Running flash begin script.... CUSTOM: Running custom begin script.... CUSTOM: Running custom begin script.... Begin script Utils/begin execution completed. Searching for SolStart directory... Checking rules.ok file... Using begin script: install_begin Using finish script: patch_finish Executing SolStart preinstall phase... Executing begin script "install_begin"... Begin script install_begin execution completed.

So far so good, it found the right stuff. Now it should set up the disks and extract my flar.

Processing profile - Opening Flash archive - Validating Flash archive - Selecting all disks - Configuring boot device - Using disk (c1t0d0) for "rootdisk" - Configuring / (c1t0d0s0) - Configuring swap (c1t0d0s1) - Configuring /GZ_VAR_LU (c1t0d0s3) - Configuring /GZ_ROOT_LU (c1t0d0s4) - Configuring /var (c1t0d0s5) - Configuring (c1t0d0s7) - Configuring (c1t2d0s7) - Configuring (c1t3d0s7) - Configuring (c1t4d0s7) - Configuring (c1t5d0s7) - Configuring (c1t6d0s7) - Configuring (c1t7d0s7) - Configuring (c1t0d0s6) - Configuring (c1t2d0s6) - Configuring (c1t3d0s6) - Configuring (c1t4d0s6) - Configuring (c1t5d0s6) - Configuring (c1t6d0s6) - Configuring (c1t7d0s6) - Deselecting unmodified disk (c1t1d0) Verifying disk configuration - WARNING: Changing the system's default boot device in the EEPROM Verifying space allocation NOTE: 1 archives did not include size information Preparing system for Flash install Configuring disk (c1t0d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t2d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t3d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t4d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t5d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t6d0) - Creating Solaris disk label (VTOC) Configuring disk (c1t7d0) - Creating Solaris disk label (VTOC) Creating and checking UFS file systems - Creating / (c1t0d0s0) - Creating /GZ_VAR_LU (c1t0d0s3) - Creating /GZ_ROOT_LU (c1t0d0s4) - Creating /var (c1t0d0s5) Beginning Flash archive processing Predeployment processing 16 blocks 16 blocks 16 blocks No local customization defined Extracting archive: patchedflar Extracted 0.00 MB ( 0% of 1302.60 MB archive) Extracted 1.00 MB ( 0% of 1302.60 MB archive) Extracted 2.00 MB ( 0% of 1302.60 MB archive) ... Extracted 1302.00 MB ( 99% of 1302.60 MB archive) Extracted 1302.60 MB (100% of 1302.60 MB archive) Extraction complete Postdeployment processing No local customization defined Customizing system files - Mount points table (/etc/vfstab) - Unselected disk mount points (/var/sadm/system/data/vfstab.unselected) - Network host addresses (/etc/hosts) - Environment variables (/etc/default/init) Cleaning devices Customizing system devices - Physical devices (/devices) - Logical devices (/dev) Installing boot information - Installing boot blocks (c1t0d0s0) - Installing boot blocks (/dev/rdsk/c1t0d0s0) - Updating system firmware for automatic rebooting Installation log location - /a/var/sadm/system/logs/install_log (before reboot) - /var/sadm/system/logs/install_log (after reboot) Flash installation complete Executing JumpStart postinstall phase... Executing finish script "Utils/finish"...

YAY!!! Still going well. Now we'll see if the JET stuff actually works. At this point, we see lots of information about moving the scripts around, copying stuff in, and setting up for the reboots and activities that happen between those reboots. After the first reboot, we see:

Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@0,0:a File and args: SunOS Release 5.10 Version Generic_127127-11 64-bit Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. myhost /dev/rdsk/c1t0d0s4 is clean /dev/rdsk/c1t0d0s3 is clean JumpStart (/var/opt/sun/jet/post_install/S99jumpstart) started @ Thu Jun 19 09:52:42 MST 2008 Loading JumpStart Server variables Loading Client configuration file Running additional install files for reboot Platform/1 NFS Mounting Media Directories Mounting nfs://10.68.75.23/export/install/patches on /var/opt/sun/jet/js_media/patch Mounting nfs://10.68.75.23/export/install/pkgs on /var/opt/sun/jet/js_media/pkg CUSTOM: Running 001.custom.001.set_etc_system SDS: Running 002.sds.001.create_fmthard SDS: Running 002.sds.001.set_boot_device SDS: Running 002.sds.002.create_metadb SDS: Running 002.sds.003.create_root_mirror SDS: Running 002.sds.007.create_user_devices SDS: Running 002.sds.008.create_filesystems SDS: Running 002.sds.009.create_vfstab_entries NFS Unmounting Media Directories Unmounting /var/opt/sun/jet/js_media/pkg Unmounting /var/opt/sun/jet/js_media/patch Save existing system entries Copying file /etc/system to /etc/system.prejs2 Jun 19 10:11:03 myhost reboot: rebooted by root Jun 19 10:11:04 myhost syslogd: going down on signal 15 Jun 19 10:11:04 rpc.metad: Terminated syncing file systems... done rebooting... Resetting...

YAY!!! Our disk configurations all executed without errors or warnings. Now let's see if it comes back clean after the reboot.

Boot device: rootdisk File and args: SunOS Release 5.10 Version Generic_127127-11 64-bit Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. myhost /dev/md/rdsk/d4 is clean /dev/md/rdsk/d3 is clean /dev/md/rdsk/d1000 is clean /dev/md/rdsk/d1001 is clean /dev/md/rdsk/d1002 is clean /dev/md/rdsk/d1003 is clean ... /dev/md/rdsk/d1352 is clean /dev/md/rdsk/d1353 is clean /dev/md/rdsk/d1354 is clean /dev/md/rdsk/d1355 is clean ... [lots of other activity deleted] SDS: Running 003.sds.001.attach_mirrors SDS: Running 003.sds.002.attach_user_mirrors Disable & delete SMF tag svc:/site/jetjump JumpStart is complete @ Thu Jun 19 10:15:55 MST 2008 myhost console login:

Success.

bill.

JET and SVM, a match made in heaven...

The flar that we created earlier now has all the packages and patches (oh yeah, JET patches the system for you as well) in it that we want, and has the basic filesystems installed for us on the root disk. We implemented the root disk layout (reserving s7 for metadb in the SVM sections later) with this section of the JET template:

# Define Root disk configuration # Make sure that /var has a Live Upgrade slice # if /var is a separate filesystem. # # Mount the LU spaces to make sure that someone # doesn't come along later and use that "free space". # # s6 is the "freehog" partition to contain all free # space after the static slices are allocated # # s7 is defined later and used as a metadb space. # base_config_profile_usedisk=rootdisk. base_config_profile_dontuse="" base_config_profile_root="8192" base_config_profile_swap="32768" base_config_profile_s3_mtpt="/GZ_VAR_LU" base_config_profile_s3_size="8192" base_config_profile_s4_mtpt="/GZ_ROOT_LU" base_config_profile_s4_size="8192" base_config_profile_s5_mtpt="/var" base_config_profile_s5_size="8192" base_config_profile_s6_mtpt="" base_config_profile_s6_size="free" base_config_profile_s7_mtpt="" base_config_profile_s7_size=""

Hint number one... Put lots of comments in your template files to remind yourself (and others who come along later) why and how you did things. These template files can be rather large and complex.

This section defines the rootdisk (reserved word in JET) with a / partition of 8G on s0, swap space of 32G on s1, /var of 8G on s5, and some Live Upgrade partitions for / and /var on s3 and s4 with matching sizes (important). The metadb space for SVM will be on s7 (defined later in the template), and any leftover space will be allocated to a partition on s6, but not mounted. We will add this free space to our pile of space for use by zones later on.

# # Any devices we need to skip (spare slices on root disks etc) # # Format: c?t?d?s? # sds_skip_devices="c1t0d0s6 c1t1d0s6"

This tells JET that we don't want to use the root mirroring phase to set up the partitions that we will use later as part of the zone disk space.

# # Additional disks under JET control. Skip the # root mirror disk (c1t1) as it is defined in the # mirroring steps below. # base_config_profile_additional_disks="c1t2d0 c1t3d0 \ c1t4d0 c1t5d0 c1t6d0 c1t7d0 "

This is where comments come into play. Without this comment, you might think that the root mirror disk should be listed in "additional disks under JET control", it just makes sense. But no, that would make very ugly things happen.

Now that we have reserved the disks for JET to use, we need to layout a disk partition scheme for them. Again, we don't want to mount them, and we will use s7 later on to add metadbs for SVM to use:

# # Define layout of the additional disks to use # s6 as the freehog space. s7 will have already # been reserved by the metadb definitions below. # base_config_profile_disk_c1t2d0s6_mtpt="" base_config_profile_disk_c1t2d0s6_size="free" base_config_profile_disk_c1t3d0s6_mtpt="" base_config_profile_disk_c1t3d0s6_size="free" base_config_profile_disk_c1t4d0s6_mtpt="" base_config_profile_disk_c1t4d0s6_size="free" base_config_profile_disk_c1t5d0s6_mtpt="" base_config_profile_disk_c1t5d0s6_size="free" base_config_profile_disk_c1t6d0s6_mtpt="" base_config_profile_disk_c1t6d0s6_size="free" base_config_profile_disk_c1t7d0s6_mtpt="" base_config_profile_disk_c1t7d0s6_size="free"

That sets up all of our physical disks (except for the metadb stuff, but that will come along later). At this point, we could install the machine and make sure that all is well, and our rootdisk works as expected. The next step is to add in the SVM stuff. Let me repeat, now is a REALLY good time to stop, try things out, and make sure that your root disk and "additional disks" are configured the way you want them.

There are three basic pieces in the SVM configuration that we need to worry about. We need to set up the metadb copies, copy the configuration of the rootdisk and mirror it, and then we need to set up the leftover diskspace and put it into a big metadevice to use later for zone space soft partitions.

# Kernel options # # If you need to increase the number of metasets from the # default (4) or the number of metadevices per metaset from # the default (128), then enter the figures here. # # Please note that increasing these numbers can significantly # increase the time taken to do a reconfiguration boot, or a # drvconfig etc. # sds_nummetasets="4" sds_nummetadevices="4000"

We needed to increase the default number of metadevices from 128 to something higher. We did some testing, and making this number in the thousands didn't hurt our performance, so we erred on the side of safety with 4000.

# This variable defines where SVM will create the metadbs. # If any meta state databases are to be created, add # the c?t?d?s? number here. If you need multiple copies # (i.e. metadb -c 3), suffix the device with a : and the # number of copies. e.g. c0t0d0s7:3 # # eg: sds_database_locations="c3t0d0s7:3 c1t0d0s7:3" # sds_database_locations="rootdisk.s7:3 c1t2d0s7:3 c1t3d0s7:3 \ c1t4d0s7:3 c1t5d0s7:3 c1t6d0s7:3 c1t7d0s7:3"

In theory, according to the template comments, you don't need to specify the rootdisk or the root mirror disk in this variable. We didn't notice the comment until we already had a working configuration, and left things alone. Your mileage may vary, but we don't get warnings or errors with this configuration and everything is working fine for us.

We have set up metadb partitions in this step, and placed three copies on each metadb partition. Again, this was a part of the build specification, and traditional for the customer. That is 24 copies of the metadb, and I am not sure if I would have configured things this way if the choice was mine. Definitely do your due diligence and make your own configuration decisions wisely. Your mileage may vary.

# # This variable ensures that partitions are created to # hold the metadbs defined above. # # Specify locations in one of the following forms # s:size - creates s7 on the rootdisk # c?t?d?s:size - creates slice on specified device # sds_database_partition="s7:32 c1t2d0s7:32 c1t3d0s7:32 \ c1t4d0s7:32 c1t5d0s7:32 c1t6d0s7:32 c1t7d0s7:32

We are creating eight metadb partitions, one on each disk, 32MB each. This section also reserves the proper space on s7 of the rootdisk. We had issues with the configuration when we tried to configure the metadb partition on s7 in the section of the template where we defined the rest of the rootdisk partitions. JET is smart enough to slice s7 out for us before calculating the "free" space for s6 on the root disk.

# If the boot device alias needs setting then do it here # ie sds_root_alias="rootdisk" # This will update the boot-device filed to ${sds_root_alias} # net and add the name to devalias, removing any previous # one of the same name sds_root_alias="rootdisk" ############ # # If we do have a root mirror, then set the devalias device # to this name # sds_root_mirror_devalias_name="rootmirr" ############ # # If you are using a two disk system and are mirroring the # root device, # you may want to enable md:mirrored_root_flag in the # kernel (/etc/system). # # You should read the info doc about this and fully understand # the implications of setting it... i.e. it's not just a case # of always setting it! # sds_mirrored_root_flag="1"

Here we assign a name for the root mirror disk. This will be the alias used in the boot prom to set up the root mirror as a second bootable device. We also set the md:mirrored_root_flag in /etc/system. Definitely read the Infodoc that the template mentions and make your own decision on this one.

# # By default, the root disk will be mirrored slice by # slice; the metadevices will start with d10 for the # first slice (sub-mirrors d11 and d12), d20 for # the second slice (sub-mirrors d21 and d22) upwards. # # If you wish to use your own numbering scheme for the # metadevices, please specify them here, in the following # format- # :mirror md:sub mirror 1:sub mirror 2 # sds_device_numbers="/:d0:d91:d92 swap:d1:d11:d12 /GZ_VAR_LU:d3:d31:d32 /GZ_ROOT_LU:d4:d41:d42 /var:d5:d51:d52"

In this section, we are defining the device names for the partitions of the root disk. We were following a local numbering scheme, and this worked well for us. One interesting note here that took a couple hours to debug, apparently metadevices may be named "d0", but they can't be named d0[anything]. So d01 is not allowed. d001 is not allowed. d02843 is not allowed. Oops. I'll definitely remember that one.

# If the root device is to be mirrored, define the mirror # here (c?t?d?). # # sds_use_fmthard can be set to "yes" | "no" | "" # # If sds_use_fmthard is set to "yes", then JET will create # metadb partitions and create the metadb as defined on # the root disk. You DO NOT need to specify the root mirror # in the sds_database_partition nor sds_database_locations # variable. # # If sds_use_fmthard is set to "no" or "", then JET will # create the data partitions for you, but you will have # to populate the sds_database_* variables if you want a # metadb to exist on the root mirror. # # You MUST set fmthard=yes for Solaris 9 and above. sds_root_mirror="c1t1d0" sds_use_fmthard=yes

Wow. That section is easy. Those two lines are all it takes to mirror the root disk. Just tell JET what disk to mirror it to, and tell JET to use "fmthard" to copy the partition table. This causes the installation to run a "prtvtoc" command against the configured root disk, and then feed that output to "fmthard -s" against the disk defined in sds_root_mirror. Simple.

For our configuration though, we will add specific naming for the root disk and root mirror disk partitions and metadevice names:

# sds_metadevices_to_init="d81:1:1:/dev/dsk/c1t0d0s0 d80:-m:d81" # # Equivalent to # metainit d81 1 1 /dev/dsk/c1t0d0s0 # metainit d80 -m d81 # # This will create a one-way mirror on d80 to d81. # # Example of combined "md.tab" and "command line" syntax: # (this would all be on one line, but has been split for # clarity) # # sds_metadevices_to_init="d71 d72 d70 d81:1:1:/dev/dsk/c1t0d0s0 # d82:1:1:/dev/dsk/c2t0d0s0 # d80:-m:d81 d91 d92 d90" sds_metadevices_to_init="d91 d92 d0 d91:1:1:/dev/dsk/c1t0d0s0 d92:1:1:/dev/dsk/c1t1d0s0 d0:-m:d91 d11 d12 d1 d11:1:1:/dev/dsk/c1t0d0s1 d12:1:1:/dev/dsk/c1t1d0s1 d1:-m:d11 d31 d32 d3 d31:1:1:/dev/dsk/c1t0d0s3 d32:1:1:/dev/dsk/c1t1d0s3 d3:-m:d31 d41 d42 d4 d41:1:1:/dev/dsk/c1t0d0s4 d42:1:1:/dev/dsk/c1t1d0s4 d4:-m:d41 d51 d52 d5 d51:1:1:/dev/dsk/c1t0d0s5 d52:1:1:/dev/dsk/c1t1d0s5 d5:-m:d51"

There we have it. Those pieces of the template define the metadbs and the root disk mirroring, and setup the partitions that we will use later to create the space for our zones. Fairly painless and straightforward, and definitely easier and safer than doing all of the twiddly bits by hand. Absolutely easier than repeating those manual tasks for 100 servers too! Next entry, I'll delve deeper into the zone partition space allocation, and the soft partitioning that we used to accomplish that piece.

This, again, is a REALLY good time to stop and try the template out. At this point, we have sliced up the root disk, created metadbs, sliced up the extra disks, and mirrored the root disk. These are the really tricky parts that will break a machine in interesting ways that are more difficult to debug. If we know that these parts are working, then creating the spaces on our leftover partitions for the zones will happen on a running and (hopefully) stable system environment, making debugging much easier.

bill.

Wednesday June 18, 2008

JET is your friend...

One of the challenges in this project is keeping an eye on the evolutionary qualities of the underlying components. In other words, we know that the basic elements of disk layout, packages added, configuration files, etc. will be in flux throughout the development and test cycles. In order to streamline these activities and make them a tad safer, we decided to implement as much as possible with the Jumpstart Enterprise Toolkit (JET).

JET allows you to create template files, describing not only basic Jumpstart configurations, but many of the most common "post install" tasks that administrators do manually. Things like JASS (Solaris Security Toolkit), Solaris Volume Manager (SVM), more complex network configurations like IPMP, and many other goodies are either implemented in JET already, or easily integrated as a repeatable and automated task. In our case, with IPMP, over 200 filesystems using SVM, and some post-install tasks (dropping in config files, setting up some standard /etc/system and ndd settings, etc.), JET is definitely a great first step on our road to our final goal of doing this through N1SPS/N1SM.

I won't cut and paste my whole template file into my blog, but I will note a few of the key sections. We started with an existing JET template from the current production environment. This had the benefit of bringing along several post-install scripts that installed default configuration files for syslog, NTP, and SSH, and gave us standard and proven locations and methods for installing some of the extra software that this project required (expect, some GNU tools, etc.). Since we are working with a flar and not using the standard pkgadd method, this line in the template file (from /opt/SUNWjet/Templates by default) gives us the starting system image that we want:

# Identify flar to load # flash_archive_locations="nfs://10.68.75.23/export/flars/SPARC-App-Tier-19.flar"

That's it. Just that line, and all the stuff from our hand-built machine is installed on the new box. Of course, we still need to specify a bunch of other configuration things, like disk layout, SVM configs, etc., but I'll get to those later.

In addition to our standard flar, we want to install some changes to /etc/system, and twiddle some extra bits. This is accomplished with a script that we wrote and saved in /opt/SUNWjet/Clients/common.files called set_etc_system. We just place that script into the JET common.files directory and configure the template to run it in reboot 1 (JET installs do several reboots after the initial flar load):

# # Override default custom scripts. # custom_scripts_1="set_etc_system" custom_scripts_m="" # # Set default locale to POSIX/C to get around buggy use of "tr" # base_config_sysidcfg_system_locale=C

Our set_etc_system script isn't super complicated, or error-proof, but I am using it as a simple example. We wrote it to automate a way around a mistake that we were making, forgetting to clean out settings between flar revisions. So for the settings that we want to be there, if the setting exists, leave it alone. If it doesn't exist in /etc/system, put our setting into the file:

#!/bin/ksh set -a # set_etc_system # 1.1 - Bill Walker < bill.walker@sun.com > # Set some sane /etc/system variables if they don't exist # ############################################################### # BN=`/usr/bin/basename ${0}` # cat >> ${ROOTDIR}/etc/system << EOF * Added by Jet set_etc_system `date +%d%b%y` EOF if grep " autoup" /etc/system >/dev/null then echo "${BN} : autoup already in /etc/system..." else cat >> ${ROOTDIR}/etc/system << EOF set autoup=480 EOF fi if grep " tune_t_fsflushr" /etc/system >/dev/null then echo "${BN} : tune_t_fsflushr already in /etc/system..." else cat >> ${ROOTDIR}/etc/system << EOF set tune_t_fsflushr=60 EOF fi if grep " rlim_fd_cur" /etc/system >/dev/null then echo "${BN} : rlim_fd_cur already in /etc/system..." else cat >> ${ROOTDIR}/etc/system << EOF set rlim_fd_cur=1024 EOF fi

Simple but repetitive tasks such as this are amazingly easy with JET. Every time you find yourself doing some task more than once on an install or deployment, just take a couple minutes and create a script to do it for you, and add it to the custom_scripts list in your JET template. Things like cleaning out the SSH known_hosts entries for users, creating a complex /etc/resolv.conf file, adding a new service, making sure that some services are disabled or sending an email or message to you to let you know that things succeeded or failed can be great time savers and make deployment safer and faster.

Not much new and exciting information here if you already use JET, but I'll be digging a bit deeper into JET in this blahg later.

bill.

Tuesday June 17, 2008

Fun and Games with JET / JS / SVM / Solaris

So this project is all about deployment. We have many (dozens of) T5220 servers, loaded with memory and disks. Within those servers, we want to deploy 36 zones each. One zone will be an MQ (Sun Java System Message Queue) instance for the local application/web zones, one zone will be a "template zone" for cloning, and 34 zones will be production zones running App Server (Sun Java System Application Server) and Web Server (Sun Java System Web Server). Of course, all of these wonderful products are part of the Sun Java Application Platform Suite, a key component of JES (Sun Java Enterprise System). There, we got all those marketing links and product names out of the way.

So this deployment architecture has several key dependencies and local requirements that presented "challenges". By "challenges", I mean decisions written in stone before the project started that make those of us doing the implementation react by rolling our eyes, sighing heavily, or responding with "Ummm... are you sure you want to do that?". Some key examples of the requirements include:

• Full Root zones

• Multiple dedicated filesystems per zone

• Separate /var filesystem, global and local zones

• Live Upgrade in global and local zones

• ZFS is not an option, Solaris Volume Manager and UFS only

• Global zone deployed with all local zone filesystems created and waiting

• Local zones deployed on demand using the pre-created filesystems

• The provisioning servers (N1SPS and N1SM) will be clustered

• The whole thing must fit on the internal disk drives

Easy, right? Yeah, nothing in there really looks like a show-stopper. Maybe some stuff that isn't "de-facto standard", or "traditional", or even "the easy way to do it". Now throw in the final requirement. From the first day that you stick a Solaris installation DVD into the machine until the "final release" of the deployment framework into the test environment is 60 days. Yeah. 60. A total of 60 days to get Jumpstart, JET, N1SPS, N1SM and Solaris Cluster all up, running, configured, and production ready, integrated into the existing customer IT environment and standards. Sure, no problem.

So step one, we need a solid foundation to build this stack on. This means having a standard flar image to deploy with. We configured a Jumpstart/JET server in our development lab, and started brainstorming our best guess as to what we would need at the end of the project as far as features, functionality, and configuration. We knew that the flar image was going to be a "frequent revision" item. So step one, to reduce deployment and patching times, and to reduce exposure, "minimize" the Solaris image on the production systems. There were over 1400 packages in the image that the customer was using (SUNWCall). This included Xwindows, Gnome, audio, CDE, demo stuff, and lots of other goodies that we knew we wanted to strip out for a headless server. My cohort started stripping out packages from SUNWCall, tracking dependencies and installation order as he went. I started with the "minimum" Metaclusters (SUNWCrnet and SUNWCmreq) and worked my way up, hoping that we would meet in the middle somewhere with a minimum of conflicts.

A couple days later, we were darn close. We came up with a configuration that ran just fine, had no outstanding dependency conflicts, ran all of our target applications, and contained 327 total packages. That is about 1100 packages gone, and statistically about 70% less potential patches to add. We definitely learned alot about packaging, patching, package dependencies, package clusters and metaclusters, and just how twisty this maze of stuff is once you try to go too deep into it. Definitely not for the conservative or inexperienced. If anyone is interested in the details, tools, and methods, drop me a note and I'll try to write a bit about the gory details.

At this point, we have a running test machine, installed by hand. We want the configuration to be a repeatable and deployable object. The perfect tool in this case is "flar" (Flash Archive). We decided to make our flar evolutionary, so we didn't have to repeat the system build every time we wanted to revision a configuration file or turn on/off a service by default. We wrote a little script called "do_flar" that cleans up log files, juggles some default settings, and puts some configs into place, runs the "flar" command to create the archive, and then cleans up the system so that it is in a decent running state for us to continue work. Rebuilding the machine is then done through JET, rebuilding our master machine using the flar (a checkpoint release from the systems side). Very cool, very easy, very repeatable. More on the do_flar script later.

The next challenge was to build the disk configuration. No ZFS allowed, and we need filesystems in each local zone for root, /var, apps, and logs. We also need Live Upgrade partitions for root and /var. That makes 6 filesystems each, times 36 local zones, plus root and /var for the global zone, plus Live Upgrade partitions there, plus swap, plus metadb space for Solaris Volume Manager. Ouch. That is well over 200 partitions, and over 200 filesystems to be created. All on the 8 internal 143GB disks. Mirrored. So all of this needs to fit on about 572GB of usable disk space. Next blog entry... Squeezing a gallon of stuff into a quart sized Zip-Loc. Or for our UK friends, squeezing a bushel of barley into a peck sized boot. Or for the rest of the known metric universe, "Holy smokes! That will never fit!!". :P

bill.