Friday August 22, 2008

Wikis for Dummies...

The folks at The CommonCraft Show have produced a bunch of interesting videos in a series called "in Plain English", explaining how technology (and Zombies) work. Very Cool stuff:

Enjoy...

bill.

Wednesday August 13, 2008

Tupperware comes in sets...

Continuing where I left off, the previous blahg entries addressed installation of the Solaris 8 branded container. Those pieces covered the mechanics of the container itself. One of the key architectural decisions in this process was "where do we put the stuff?". Not just mountpoints and filesystems, we already covered that, but what pieces go on local disk storage, and what pieces go on the shared SAN storage?

Since the objective is to eventually integrate into a failover scenario, we looked at two options here. Each one has benefits and can supply a capability to our final solution. In the first case, we want to fail a container over to an alternate host system. In the second case, we want to fail a container over to an alternate datacenter. Think of these two as "Business Continuity" and "Disaster Recovery".

In the Business Continuity case, the capability to do "rolling upgrades" as part of the solution would be a huge added bonus. We decided to put the zone itself on local disk storage, and the application data on the shared SAN storage. This allows us to "upgrade" a container, roll the application in, and still maintain a "fallback" configuration in case the upgrade causes problems, with minimal downtime. Accomplishing this requires two copies of the container. Application data "rollback" and "fallback" scenarios are satisfied with the shared SAN storage itself through snapshots and point in time copies.

Similar to a cluster failover pair, both zones have their own patch levels and configurations, and a shared IP address can be used for accessing application services. Only one zone can be “live” at any time as these two zones are actually copies of the same zone “system”.

To migrate the branded container to another host system, the zone must be halted, and the shared SAN storage volumes must be detached, and unmounted from the original host system:

The detach operation saves information about the container and its configuration in an XML file in the ZONEPATH (/zones/[zonename] in our configuration). This will allow the container to be created on the target system with minimal manual configuration through zonecfg.

The detached container’s filesystem can now be safely copied to the new target system. The filesystem will be backed up and then restored on to the target system. There are many utilities that can create and extract backup images, this example uses the “pax” utility. A pax archive can preserve information about the filesystem, including ACLs, permissions, creation, access, and modification times, and most types of “special files” that are persistent. Make sure that there is enough space on both the source system and the target system to hold the pax archive (/path/to/[zonename].pax in the example) as the image may be several gigabytes in size. Some warnings could be seen during the pax archive creation. Some transient special files cannot be archived, but will be re-created on the target system when the zone boots.

On the target system, the zone filesystem space must be configured and mounted, and have “700” permissions with owner root. The /zones loopback mount must also be in place, just as in the source system.

Since the zone filesystem is not on shared storage, and will remain local to the target system, the “mount at boot” option can be set to “yes”.

Storage for the applications and data should now be imported and mounted on the target system to replicate the configuration of the source system. All mountpoints, loopback filesystems, and targets of the “add fs” components of the zone must be replicated. Once the filesystems are mounted into the global zone, the zone pax archive can be extracted. Again, care must be taken to make sure that there is sufficient space on the zone filesystem for the extraction:

The filesystem of the zone is now in place, but the zone is not yet configured into the target system. The zone must be created, modified as necessary (i.e. different network adapter hardware or device naming), and “attached” to the new host system. As a sanity check, it is highly recommended that the /usr/lib/brand/solaris8/s8_p2v command is run against the new zone to make sure that the new system “accepts” the attach of a zone created elsewhere:

The “attach” command may fail with messages about patch version conflicts, as well as extra or missing patches. Even though this is a full root zone, the detach/attach functionality makes sure that the host systems are equivalent. Some patches will be missing or extra in some cases, especially where the machine types or CPU types are different (sun4u, sun4v, HBA types and models, Ethernet adapter hardware differences, etc.). It is possible to normalize all patch versions and instances across systems of different configurations and architectures, but this involves significant effort and planning, and has no real effect on the operation of the hosting systems or the hosted zones (patching software that will never run on a given machine).

Once all errors and warnings are accounted for as “accepted deltas” or resolved, a failed attach can be forced:

Zone migration can be toggled between the machines by halting the zone, detaching the zone, moving the shared SAN storage into the target system, attaching the zone and booting the zone. Once the zone has been installed, configured, and booted on both systems, there is no need to use the s8_p2v function for migration. Strictly speaking, the “detach/attach” function is not necessary since the zone itself resides locally, and is not actually migrating, but it does provide an extra layer of protection on the non-active machine to keep the halted zone from being booted while the shared storage is not active. By setting the zone state to “detached”, the zone will not boot unless the “attach” command is executed first, providing the check for the shared SAN storage configured with the “add fs” zone configuration.

Pretty simple, huh? In fact, if you look at the above diagram, it looks mysteriously like the functionality of a cluster failover. Once we modeled and tested these actions by hand, we integrated the pair of containers into Veritas Cluster Server and managed the zones through the VCS GUI. Online, offline, failover... It all just works. Very cool stuff.

bill.

Tuesday August 12, 2008

Tupperware footnote...

A couple folks have emailed me asking about my VxVM and VxFS in this physical to virtual conversion. As I have blahg'd before, separation is a good thing. In this case, we had Veritas Volume Manager and Veritas Filesystem on the source machine, and on the target machine. Volume Management and Filesystem Management should live within the Global Zone, not in local zones (or non-global zones as they are officially called). Mixing "system" activities within zones is a BadThing[tm], especially when the zone is a branded container. Trying to use Solaris 8 filesystem utilities and disk volume management utility binaries (even through the branded containers software) against a Solaris 10 system, kernel, and possibly even a hardware architecture (sun4v) unknown to a Solaris 8 operating system is a dangerous path to walk.

Definitely too much risk in there for me to even attempt to whack it into working.

We installed out Volume Management (VxVM) and Filesystem (VxFS) on the Solaris 10 target host system using the Veritas Foundation Suite (5.0 maintenance pack 1 Rolling Patch 4). All of the storage goodies were installed and configured as local objects on the Solaris 10 host system, and mounted under the /z/[zonename]_[function] pathnames as described earlier. The lofs loopback mounts and zonecfg "add fs" pieces mapped them into the places that we wanted them to be, just providing "disk space" to the Solaris 8 branded containers. We did use zonecfg "add fs" with a type of vxfs, and it worked as advertised. In the end, we decided that the VxFS pieces are a "system function" and should be mounted in the Global Zone under /z for simplicity and consistency.

Who knows, at some point we might even use ZFS instead of the VxFS in this configuration (more on that in a later blahg entry), and this allows us to keep the "zone space" filesystem and storage agnostic.

bill.

Monday August 11, 2008

Tupperware isn't the only Branded Container in town...

For this project, two Solaris 8 branded containers were installed on the test systems using "flar" images. The containers were created “unconfigured” so that IP addresses could be assigned to avoid conflict with the source systems. Instructions from the docs.sun.com Solaris 8 branded containers administration guide were used to install and configure the test containers. The docs in the administration guide were excellent, with lots of "type this" and "you should see this" guidance.

We had two test scenarios that we wanted to play with: (1) Branded containers on SAN shared storage, and (2) branded containers hosted on local storage with SAN shared storage for application data. There are advantages and disadvantages for both, and situations where each has significant value. I'll get into the details of that in a future blahg entry.

In order to separate the "storage administration" from the “zone administration” function, storage was configured with device mounts within /z, regardless of the type (SAN, NAS or internal) of storage being utilized:

The storage for the container was mounted to the /zones/[zonename]/ path using loopback filesystems (lofs). This mount was created for testing using the command line:

So that the mount could persist for reboots, an entry was added to /etc/vfstab:

Note that the “mount at boot” option is set to “no” in this example. This is to allow for zones installed on SAN shared storage volumes to migrate back and forth between the systems. Zones installed on local, internal storage will use “yes” for the “mount at boot” option. In the shared SAN storage zones, the filesystems must be mounted as part of the zone management when rebooting, or attaching a zone into a new host:

The basic Solaris 8 branded container was created with the zonecfg command on the first Solaris 10 host system. The basic Solaris 8 branded zone contains a zonepath for the zone to be installed within, and a network interface for network access:

Extra filesystems for administrative tools and applications are mounted into the branded container from the global zone mountpoints with the “add fs” command within zonecfg. The loopback filesystem (lofs) is used to allow the Solaris 10 global zone to manage devices, volume management, filesystems, and mountpoints, and “project” that filesystem space into the branded container. While it is possible to pass physical devices in to a container, it is not a recommended architecture when at all possible to avoid, especially in the case of branded containers, where device and filesystem management would be running under the brand, while the kernel being leveraged would be of a different OS release. This loopback filesystem is defined during the zone configuration with:

Because the Solaris 10 Global Zone (GZ) can run on machines that don’t support native Solaris 8, some applications and software packages could become confused about the architecture of the underlying hardware and cause problems. The Solaris 8 branded container is shielding us from the hardware platform hosting the Solaris 10 GZ, so it is recommended that we set a machine architecture of “sun4u” (Sun UltraSPARC) for the branded container so that the hardware platform is essentially hidden from the operating environment within the container. The “machine” attribute can be assigned within the zone configuration with:

Other filesystems can be added, additional network interfaces can be defined, and other system attributes can be assigned to the branded container as needed. Once the container has been configured within zonecfg, the configuration should be verified and committed to save the data.

The branded container is now configured and ready for installation of an image (flar) from a physical system. The Solaris 8 system image is created using the flarcreate command. Make sure that the source system is up to speed on patches, especially those related to patching, flash archives (flar) and the "df" patch (137749) that I mentioned in an earlier blahg entry.

The configured zone can now be installed using the flar image of the physical Solaris 8 based system:

Once the zone has been installed, it the “P2V” utility must be run to turn the physical machine configuration into a virtual machine configuration within the zone:

The Solaris 8 branded container can now be booted. There will likely be some system startup scripts, tools, and applications that will give warning messages and errors on boot up. Remember the nature of the zone / host system relationship and decide what needs to run in a zone and what functionality should remain in the host system global zone. The zone will boot “un-configured”, and ask for hostname, console terminal type, name service information, and network configurations on the first boot. As an alternative, a “sysidcfg” file can be copied into the /zones/[zonename]/root/etc/ directory before the first boot to allow the zone to auto-configure with sysid upon first boot.

That's it. Other than fixing up the system startup scripts (/etc/rc*.d and /etc/init.d), and attaching in a copy of the source system's SAN attached data, the move is done and ready to be tested. The really cool part of this is that it just works. We were expecting some application issues and possibly some "speed of light" problems, but everything just worked. Obviously some things had to be adjusted in the branded container, disabling the Veritas Volume Manager startup, and some hardware inventory scripts that used "prtconf" to collect information, but these were identified early, and several reboots sorted out the symptoms from the zone's boot messages on the zlogin console.

More details later about migration of the zone between systems, various storage configurations that we tested, and some other (hopefully) interesting thoughts.

bill.

Monday July 28, 2008

Separation is a Good Thing...

One of the interesting side effects of using containers/zones is that it provides a layer of separation for administration functions. Along the way, we have an opportunity to look at other separations that we can provide between the various IT organizations contributing to the "service" being provided by the system.

As an example, how many of us IT types here "I need the root password" from the database or application folks on a regular basis. Well, unless you have implemented RBAC (Role Based Access Control), and have trained your developers and DBAs to the point that they know what to ask for, this is probably a common request. I am alot more comfortable as an administrator giving administrative access to a zone than I am in giving it within the Global Zone. I know that if they totally roach things in the local zone, I can still edit files and move things around from the Global Zone to fix things up, and the system is not "down" or in a state that I need to boot from DVD or Jumpstart and twiddle bits.

Implementing zones gives us a chance to integrate several areas of separation, providing simpler administration. One example, I like to separate "disk/storage administration" from system/zone administration. I mount the (non-ZFS) storage under /z and then use the loopback filesystem (lofs) to make my zonepath. for example:

/z/[zonename]_root lofs mounted on /zones/[zonename] /z/[zonename]_opt_oracle zonecfg "add fs" with mountpoint=/opt/oracle /z/[zonename]_data_oracle zonecfg "add fs" with mountpoint=/data/oracle

Obviously this is even easier with ZFS, but I might get to that another day in another blahg entry. Using the resource mountpoint and the zonename in the filesystem information allows me to use grep in interesting ways with "df" and "mount" to more easily track down where things are being mounted and used.

I also try to avoid using devices within zones. This is especially important when I am using branded containers, as it creates interesting dependencies between the Solaris 8 software and the Solaris 10 kernel interfaces. One prime example, Veritas Filesystem (vxfs) and Veritas Volume Manager (vxvm). Often, when migrating a physical Solaris 8 machine into a branded container, admins will try to move administrative functions into the container and treat it as a real machine. There are huge benefits in moving "administrative functions" into the Global Zone, and leaving the application functionality within the branded container. Just say no. Manage your local storage, SAN, and volume resources from the GZ, and just present "disk space" into the local zones.

Other administrative functions that are much easier to keep in the GZ include backups, network administration, configuration management, and performance and capacity management. I find alot of effort being spent on trying to "shoehorn" these functions into a container, when logically they belong in the "system", or GZ. Separating the "system" (there is only one, and it is a shared resource) and the "application" (there can be many, and they consume the "system) has huge benefits in the ongoing administration and maintenance efforts. This is more of a "how to think" problem than a "what to think" education.

Just my opinions, your mileage may vary, objects in mirror may be closer than they appear, etc.. Feel free to comment, debate, or object in the comments below.

bill.

Saturday July 26, 2008

Branded Container Hurdle...

We ran into an interesting hurdle with our branded containers project. Once we had everything up and running, we decided to test patching the global zone (Solaris 10) and patching within the local zone (Solaris 8). We downloaded both "Recommended Patch Clusters" from SunSolve, and applied them.

The Solaris 10 patch installation went without a hitch (as expected), but adding patches to the Solaris 8 branded container failed. The error message from patchadd was that there wasn't enough disk space to install the patches. None of the patches would install. We did some checking, and "df -h" showed that we had 22GB of free disk space on the partition containing the zone. Weird. So we looked at patchadd, and it was running "df" on the /var/sadm directory to see if there was enough space to hold the files necessary to back out the patches. "df -k /var/sadm" failed with a message "special device and mount point are the same for loopback file system". So "df -k" worked fine, but "df -k [something]" would fail. Interesting.

I popped a note off to the Branded Containers team at Sun, and received an answer within 10 minutes. Apparently I was running into a known (and fixed) bug:

Status: RELEASED Patch Id: 137749-01 fixes BugID 6653043 ... Description: When running A solaris 8 zone within Migration Assistant (etude), it is possible to have a loopback filesystem mount in which the underlying mount is not visible in the zone. The actual issue is addressed very simply in the Solaris 10 df by saving the best match while doing the walk backwards through the mount table.

This bug manifests itself if your branded zone is using filesystems that have loopback filesystems (lofs) underneath. Since we were mounting our disk space under /z/[zonename]_root and then using lofs to mount it into the zonepath (/zones/[zonename]), this one bit us. Even our application and data spaces were mounted under /z (with zonename and resource name included in the mountpoint name) and then imported into the zones with zonecfg "add fs". A little more research revealed that the error message that we were receiving was actually removed from the Solaris 8 source code as part of the fix.

- if (EQ(match->mte_mount->mnt_fstype, MNTTYPE_LOFS) && - EQ(match->mte_mount->mnt_mountp, - match->mte_mount->mnt_special)) - errmsg(ERR_FATAL, - "special device and mount point are the same for loopback file system");

Now comes a more interesting question, if I need a patch to make adding patches work, how can I install that patch? The answer is to use the "patchadd -d" option to tell patchadd not to try and save the old versions of the files being patched (in this case, just the "df" binary).

Interesting note, patch 137749-01 is not contained in the latest Solaris 8 recommended patch cluster, or in the Solaris 8 Branded Containers package. This patch must be downloaded separately and applied by hand to the Solaris 8 branded container.

One of the Branded Containers support team guys did file an RFE to get the patch included in the software distribution for Solaris 8 Branded Containers for a future release.

Hopefully there is enough info in this blahg to help others who might Google the error message or symptom, and provide a quick path to the answer.

bill.

Friday July 18, 2008

Branded zones are your friend...

I'm working with a different customer for a couple of weeks. This is a large financial/insurance company going through a fairly common set of issues. The primary issue that I am here to help with is that alot of their environment is running on Solaris 8, and they are finding it very difficult to justify buying "old school" hardware to expand the Solaris 8 server farm. Until they can update the applications and complete testing cycles, the new hardware and OS features aren't an option. Until now.

We are doing Physical machine to Virtual machine (P2V) work to re-host some of those Solaris 8 workloads into Solaris 8 branded containers (zones) on a Solaris 10 host. Lots of advantages here. We can now run on up to date hardware (testing now with a T5220). We can take advantage of ZFS, Dtrace, performance improvements, and all those other Solaris 10 features that didn't exist in Solaris 8. Best of all, we can use the zones as a development, test, and migration tool moving forward to bring these working environments up to current releases of software applications, tools, and operating environment without having to spawn off even more machines for the migrations. The applications running in Solaris 8 branded zones now on host X will become Solaris 10 native zones at some point, running the updated applications and services.

There are several blahg entries to come... Doing the P2V to host the Solaris 8 "system" into a branded zone. Migrating the branded zone back and forth between physical machines (think clustering and hardware upgrades/service windows). Integration of this work within a SAN environment with BCVs (for backup services) and volume management. Cloning / copying the production zones into development and test environments. There are tons of possibilities in this kind of architecture, and a few gotchas and constraints to go along with them. I'll cover some of the key points over the next week or so.

Since I don't have a SPARC machine in my hotel room, I created "Solaris 10 branded containers" on x86. This enables me to play around under VirtualBox on my laptop from a hotel room to model and test. Yeah, this isn't a supported function at all, but it did enable me to learn alot about how zones (and particularly branded zones) work. Key features here include being able to install a zone from a flar or ufsdump image of a physical system or VirtualBox VM (P2V for Solaris 10 x86) from the zonecfg interface, and being able to emulate the SAN "attach/import/export/detach" functionality by moving my virtual disks that contain the branded zones between virtual machines.

Yeah, alot learned this week, and not enough time to write up the details just yet.

So here is is, the end of the first week, and we have:

• Installed two systems with Solaris 10, patched

• Installed the branded containers software

• Created, mounted, and configured our SAN based storage

• Created, configured, installed, and verified a pair of Solaris 8 branded containers from production system flars (unconfig'd and preserved)

• Used zonecfg detach / attach and appropriate storage magic to move zones between the physical machines

• Learned *alot* about how zones and branded zones function with the global zone, device mounts, storage devices, etc..

Wow. All that in a week. This team rocks. Oh yeah, so does VirtualBox, Solaris 10, zones, branded zones, docs.sun.com and my favorite debugging tool, Google.

bill.

Friday July 11, 2008

Patches?! We don't need no steenking patches...

"Badges? We ain't got no badges. We don't need no badges. I don't have to show you any stinking badges!" --Gold Hat, as played by Alfonso Bedoya "The Treasure of the Sierra Madre" (1948)

I have been using my Virtualbox hosted JET server to install my VMs in Virtualbox for a couple weeks now. I have been using Solaris 10 update 5, and haven't worried about new patches. Until now.

Apparently the JET package from BigAdmin expects patch clusters/bundles to be in a directory called "10_x86_Recommended" with a patch_order file sitting amongst the individual patch directories (i.e. 109773-10/). The new patch clusters from SunSolve (using the 0508 extracted zips as an example) look like:

 patchbundle_0508_x86/
      Copyright                  
      README
      Copyright_chunk2           
      installbundle.sh*
      LEGAL_LICENSE.TXT          
      patch_order
      LEGAL_LICENSE_chunk2.TXT   
      patchbundle.conf             
      prereq_order
      Patches/                   
            120411-28
            120413-08
            119255-53
            127885-01
            113000-07
            ...

The three options are pretty simple. We could run the installbundle.sh as a post-install task after the first reboot in this section of the template. Don't forget to include an NFS mount of the patch bundle directory in your custom script, and read the README file to make sure everything will work. The install_bundle.sh script requires a password to install to make sure you have read the README.

############ # # Scripts to be run on the client at the end of the build # # The scripts must be placed in the directory # /opt/jet/Clients/ # and will be copied to the client. # # If you want to run custom scripts during the Jumpstart # phase, use the custom_scripts_f variable below. # # Custom scripts at subsequent boots # # denotes the boot number you want the action to be performed. # You can create new variables for boot levels 2,3,4 etc. # n means after the last reboot. i.e. last. # m means n-1. i.e. before the last reboot. Use m if you need to # guarantee a reboot after the action is performed. # custom_scripts_1="" custom_scripts_m="" custom_scripts_n=""

The second option would be to define the patch bundle as a custom patch set in this section of the template:

# # Custom patch sets... create a directory in the patch # directory named after the set, and put a patch_order # file in it, along with the patches... # (Space seperated list of patch set names) # # N.B. as a side effect, if a directory exists under the # patch set dir named # after the OS, (uname -r), the subdirectory will be # used instead of the main patchset directory # i.e /export/install/patches/patchset/5.8 takes # preference over /export/install/patches/patchset # custom_patchsets=""

The third option (that I chose) was to make my patch tree appear to be what JET was expecting, using this section of the template:

############ # # N.B. Unless you need to point this client at alternate # media for patches and packages that is not held on # this server, please skip this section! # # productdir is where to find the products. This should # be a URI style path, i.e. # nfs://192.168.1.1/export/install/pkgs. # If the server is the JumpStart server, then # it should just be specified as a normal path. # # patchdir is where to find the patches. Same format # as productdir. # base_config_productdir="" base_config_patchdir="/export/install/patches"

In order for this to work, the patches have to be in the "10_x86_Recommended" directory (discovered through trial and error, and looking at the logs on the installed VM), accompanied by a patch_order file.

ayrton# cd /export/install/patches ayrton# ls patchbundle_0508_x86 ayrton# ln -s patchbundle_0508_x86/Patches 10_x86_Recommended ayrton# cd patchbundle_0508_x86/Patches ayrton# ln -s ../patch_order patch_order

So now there is a 10_x86_Recommended directory (symlink) visible in /export/install/patches that contains the patch directories and a patch_order file. As always, your mileage may vary, no warranty expressed or implied, but it worked for me.

PS - Kudos to Mike Ramchand for finding my real answer. Apparently the 0508 bundle is supposed to bring you up to the level of the 05/08 Solaris update release. The real patch cluster (with the correct directory structure) is on SunSolve under "Solaris 10 x86", buried down in the list underneath the "Solaris 10 x86 0508 Patch Bundle" entry. Oops. Oh well, now we have two choices, but the obvious "best answer" is to just install the update 5 Solaris image in the first place from your JET server.

bill.

Saturday July 05, 2008

Happy 4th Everyone!!

Great to have a weekend off with the family.

Guess how we spent our time?

bill.

Thursday July 03, 2008

xVM Ops Center in a Virtual world...

Yet another little homespun project that will make people ask "Why?!". I need to get up to speed on xVM Ops Center to stay ahead of my customers. It also seems like a really cool tool for managing the dozen or so machines around my house. Yes, I have three kids, 9, 4, and 2, and one computer literate wife. Yet there are about a dozen machines up and running at any given time, plus a pile of virtual machines serving a variety of purposes. There are SunRays around the place, great gadgets for ordering car parts from the garage if you don't want to keep running inside and washing your hands to use a computer. Those spill-proof keyboards like the ones they use at Jiffy Lube rock for greasy fingers searching for 30 year old wiper switches on eBay.

Here is a little screenshot from within a xVM VirtualBox Solaris 10 VM on my desktop... Let your mind wander to the possibilities.

I'll bet you thought of xVM Ops Center as a potentially fabulous tool for managing all of your servers and possibly even virtual machines within those servers. But why not run the Ops Center itself from within a virtual machine? Hmmm... Think on that one for a while, and I'll be back with more details.

bill.

VirtualBox with a promiscuous floozy...

Everything was working just fine on my laptop machine, my VirtualBox VMs could see each other, Jumpstart and JET were just fine, and all of the VMs could see the outside world. When I moved my configs into my ridiculously over-configured desktop machine (see blahg entries from about 18 months ago), networking stopped functioning. Weird. In fact, my Windows host machine stopped communicating to the outside world as well. Ouch.

VirtualBox uses a "TAP" interface driver to bridge or NAT the VM networks through the WIndows host. On the Windows side, in order for this "TAP" adapter to talk to the outside world, it must use the "Network Bridge" or "MAC Miniport Adapter". If you select the TAP adapter, CTRL-click your network adapter that talks to the outside world, right mouse click and select "Add to Bridge", the two adapters are linked together into a single logical interface. Packets from one side can magically appear on the other side. In fact, you should be able to "snoop" from a Solaris virtual machine and see packets flying around on your external network. The final Windows XP Network Connections window should look something like this:

Yeah, that's how it is supposed to work. Mine didn't. I checked the Firewall settings, and even disabled it temporarily to make sure that it wasn't the problem. I Googled (and yes, that is a verb), and found nothing apropos. After several hours of head scratching, reading blogs, reading through the Microsoft technical docs, and banging my head against the keyboard, I stumbled on the answer. My adapters were not in "promiscuous mode". Promiscuous mode allows adapters to pass through all of the packets that they see, rather than just the packets that are addressed to this host. That sounds like what I need.

The magic mojo to make my bridged adapters go into promiscuous mode was fairly painless. Click on [Start] -> [Run] and type "cmd" to get a shell window on the XP host. In the shell window, I was able to run the netsh commands necessary to make the adapters go promiscuous, and to check that the changes were in place. This changes registry entries, so you should be able to just jump through these hoops once, and the settings should remain through reboots:

Microsoft Windows XP [Version 5.1.2600] (C) Copyright 1985-2001 Microsoft Corp. E:\Documents and Settings\Bill Walker>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- ---------------------------------------------------------------------- E:\>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- 1 VirtualBox Host Interface 1 unknown 2 Local Area Connection 2 unknown ---------------------------------------------------------------------- E:\>netsh bridge set adapter 1 forcecompatmode=enable E:\>netsh bridge set adapter 2 forcecompatmode=enable E:\>netsh bridge show adapter ---------------------------------------------------------------------- ID AdapterFriendlyName ForceCompatibilityMode ---------------------------------------------------------------------- 1 VirtualBox Host Interface 1 enabled 2 Local Area Connection 2 enabled ----------------------------------------------------------------------

Voila! My Windows network is back to a sane mode. My VPN connections to the office are alive and well. My VirtualBox VMs can now see the outside world. I now have my Frankenstein super-desktop (TBs of disk, 8G of memory, 4x Opteron cores) back in a happy state. As always, your mileage may vary, objects in mirror may be closer than they appear, no warranty expressed or implied, but I hope this helps someone save hours of head banging on fragile keyboards at some point.

bill.

Sunday June 29, 2008

VirtualBox, the VM JET Client Chronicles...

Here is the JET template for my simple, sample virtual machine example. Wow, I've been watching way too much toddler TV. I'm starting to rhyme things for no apparent reason.

# Ethernet can be obtained from the 'banner' command at OBP # # OS is one of the values you used to register the solaris media using # the add_solaris_location command # base_config_ClientArch=i86pc base_config_ClientEther=08:00:27:8C:D9:E0 base_config_ClientOS=S10u5

This gives us the ethernet address, chip architecture, and OS version to be used to install the client VM.

# # Client allocation # # The mechanism used to build this client; by default, the options listed # in /opt/jet/etc/jumpstart.conf will be tried; you should only set this # if this particular client needs to do something different. # JET supports bootp, dhcp, and grub as allocation options. # Currently grub is only supported on i86pc architectures. # base_config_client_allocation="dhcp"

Using "dhcp" allows me to do the PXE boot magic inside the VM intnet network.

############ # # JumpStart sysidcfg information # base_config_sysidcfg_nameservice=NONE base_config_sysidcfg_network_interface=PRIMARY base_config_sysidcfg_ip_address=192.168.1.202 base_config_sysidcfg_netmask=255.255.255.0 base_config_sysidcfg_root_password="JGnRL6GHcobuc" base_config_sysidcfg_system_locale="C" base_config_sysidcfg_timeserver=localhost base_config_sysidcfg_timezone="US/Eastern" base_config_sysidcfg_terminal=vt100 base_config_sysidcfg_security_policy=NONE base_config_sysidcfg_protocol_ipv6=no base_config_sysidcfg_default_route=192.168.1.1

This is the standard sysidcfg stuff. Same here as in Jumpstart, or bringing up a sys-unconfig'd machine.

# base_config_x86_nowin="yes" base_config_x86_console="" base_config_x86_disable_acpi="" base_config_x86_safetoreboot="yes" base_config_x86_disable_kdmconfig="yes" base_config_x86_confflags="" base_config_sysidcfg_x86_kdmfile=""

Some simple options for booting and setup. The comments and documentation explain all of these fairly well.

base_config_profile_cluster=SUNWCall base_config_profile_usedisk=rootdisk. base_config_profile_dontuse="" base_config_profile_root=free base_config_profile_swap=256

This is the "what to install" and the simple root disk configuration. I am reserving 256M for a swap partition, and putting the rest of the available space on the root partition.

That's the basics. You can do alot more with Template files, including the SVM and JASS modules. Read the comments in the make_template generated templates, read the docs, read the blogs, and do lots of copying and pasting.

There we go. We have now created a totally hands-off installation of a system inside the VirtualBox. If we configured the boot devices correctly (disk first, network second), then there was no interaction necessary between clicking the "Start" button on the VirtualBox GUI, and getting a login screen on the fully configured and running client system. Very cool stuff.

bill.

Saturday June 28, 2008

VirtualBox meets JET...

So first, the why. I was working on a project (described in detail in earlier blahg entries) that used JET/Jumpstart to maintain "golden images" of systems for provisioning in the application and web tiers. First reason for doing this was just to have a JET configuration that I could play with in my hotel room while waiting for room service. Sitting around the office behind the firewalls just to get access to the boxes can be a bummer after about 8pm.

Second reason for this, I'm a geek. Once I had the idea in my head, I just had to get it to work. Third reason, it is a really handy way to set up machine variances within VMs on your desktop or laptop for demos and testing software. Yeah, you can "clone" VMs, and copy the disk images and configs around. But you never know when you are going to "fat finger" a VM, or forget some important step when munging things by hand. Using a JET server allows you to save a config, re-use it, and always start with a fresh "machine" to work on, in a known state. You can even save off your favorite VMs as "flar" images to restore on demand, using compressed images can save *tons* of space, especially important when you are on a disk-limited road warrior laptop system.

Step one, download Sun xVM VirtualBox and read all of the assorted docs and technical info. Join the community, read the Wikis, read some blogs.

Step two, install the downloaded package on your system of choice. In my case, I am running it on my laptop and desktop machines, both under Windows XP. There are lots of folks running other host systems, read the info, read the blogs.

Step three, download your Solaris or OpenSolaris DVD image of choice. No need to burn the image on to DVD media, we'll just mount the ISO DVD image as a virtual DVD later. While you are cruising OpenSolaris.org, join some discussions, cruise through the forums, read some documentation, join the movement. (Can you tell I really like OpenSolaris?)

Step four, (well, maybe after you play around with VirtualBox a bit and get comfortable with it) create your Jet Server virtual machine (VM). Double click on the desktop icon, and then click the "New" button. For my configuration, I went with an 32G root disk, added in 32G for /export (only allocated from your hard drive as you use the space), 512Meg of RAM (I have 8Gig on my desktop and 4Gig in my laptop), and defined the CD drive to be a virtual drive, mounting the ISO image that we just downloaded.

The networks here are the tricky part. I am using a "Back End Network" (BEN) for management. I will keep my JET/Jumpstart traffic on the BEN, and use a second interface as a "public network". My BEN network is defined as "internal network" or "intnet" (inter-VM traffic only). My external interface could be NAT or host bridge, either way, the Windows host can get to the VM, and the VM can see the outside world (Firefox can get to Google). I configure the VM through the GUI, but then ran into a little "issue".

There was a bug in the earlier (1.3.X) version of VirtualBox that broke "intnet" network connections. A couple packets would fly back and forth between VMs, and then the connections would just disappear. You could snoop and see packets going out, and sometimes being received, but answering packets never made it back to the originating host. I don't know if that is fixed in the current version (1.6), but there is a workaround. In the Windows host, click [Start] -> [Run], and then type in "cmd" to get a shell window. Use "cd" to go to the VirtualBox installation directory (C:\Program Files\Sun\xVM VirtualBox in my case), and use the command line utility VBoxManage.exe. The magic mojo here is (for my VM named JETserver):

  "C:\...xVM VirtualBox> VboxManage modifyvm JETserver -nic1 intnet

Note that the network interfaces are "nic1" (my BEN) and "nic2" (my public interface).

Now just click the "Start" button, followed quickly by pressing "F12" on your keyboard to choose the boot device. This option only appears during the splash screen of the virtual BIOS, so you have a second or two to hit the key. Choose CDROM for your boot device and away you go installing Solaris as usual. You can also juggle the boot devices under [General] -> [Advanced] -> [boot devices] in the configuration pane. Make sure that if you created a second drive for the JET stuff (as I did) that you set it up during the install, or fdisk/format/newfs it when the machine is booted. Make sure that you set your boot devices again, or unmount the virtual CDROM device, or hit F12 really quickly again to boot from disk after the install.

There we go, a Solaris VM. Yay! Now we need to make it into a JET server.

Step five (OK, so four was rather long), download the Jumpstart Enterprise Toolkit (JET) software from the JET home on BigAdmin. BigAdmin is your friend. Cruise around, read the JET Wiki, read docs, join communities, etc. as usual. Also take a cruise through Mike Ramchand's blog, and check out the Yahoo! JETJumpStart group.

Step six, follow the instructions for installing JET, using that ISO file again as a virtual CDROM device. Make sure you run:

/opt/SUNWjet/Products/base_config/solaris/make_dhcp 

to get the DHCP and PXE boot stuff in place, since we will be using those goodies to boot our client VMs.

The documentation for JET is included in the package in PDF format. Read it (at least give it a cursory pass through) before continuing. The simple stuff is really simple. The more complex and intricate operations can get tricky.

Step seven, create a new VM to be a JET client. This is simple, here is a screenshot of my JSclient machine to show the settings (don't forget to jump out to the command line again for VBoxManage.exe after you configure it through the GUI):

Take note of the ethernet address of your virtual adapter, and use that in your JET template. Make sure you assign the IP address of your JET client in the template file to be on the same network as your BEN "intnet" connection on the JET server VM. If you want to get fancy, you can add a public interface to the JET client configuration and add that to the JET template as well (DHCP or fixed IP with default router and DNS server info in the template).

Step eight, boot the sucker, hit "F12", and choose network boot (PXE) (or again, juggle your boot devices under the [General] -> [Advanced] -> [boot devices] configuration pane). If things don't work as planned, you can snoop your nic1 from the JETserver VM to debug things. That's it. Simple. I'll post some more screen shots and details of my template files as time permits.

bill.

Friday June 27, 2008

It's a Digital World (and then some!)...

Kudos to my friend Alok Rishi for presenting us with this evidence of blue collar digital overload from Hong Kong.

Wow. I especially like the pink one with the tassels. I wonder if that is the new quad-band Barbie phone?

bill.

Strawberry Fields Forever...

I took the family down to Delaplane, VA for the local "Strawberry Festival". It was actually held at the Sky Meadows State Park about 7 miles north of Delaplane. There were plenty of food vendors and kitschy craft stuff there to keep us emptying our wallets. One booth had a ton of tie-dye goodies including t-shirts, hats, handbags, diaper bags, and just about anything else that is made of cotton and can be tied and dipped. I got a couple cool shirts there.

Definitely a fun place for the kids, as they had hay rides, obstacle courses made of hay bales, and tons of crafts that the kids could do for a couple bucks. My kids all made sand art bottles with lots of neon colored sand. It was definitely a blast for them, and not too expensive. The food choices were plentiful, from fish and chips to hot dogs and italian sausage, to funnel cakes. There were even a couple ice cream vendors making hand dipped strawberry sundaes on shortcake, or a choice of dozens of flavors on a cone.

My personal favorite vendor brought back some fond memories of my past. I attended Penn State Mont Alto back in the mid 80's. A couple miles up the road was this really cool place called "Mr. Ed's" (more info on Mr Ed's here). Mr Ed's is a peanut lover's heaven, a candy lover's heaven, has great fudge, and tons of cool toys and stuffed animals for kids. Mr. Ed's had a stand at the festival and was roasting peanuts fresh on site. The smell brought back fond memories for me, and I had to buy a couple pounds of peanuts and some old-school candy for the kids. We even made a road trip to Mr Ed's store on US Rt 30 (Lincoln Highway) with the kids a couple weeks later.

Of course, any town fair, carnival, or rural strawberry festival isn't complete for my children until they get to ride a pony. This one was no exception. In addition to the 4-H sponsored petting zoo, they had pony rides for a couple bucks, with a bad Polaroid picture included.

Samantha the Princess:

Shelby the other Princess:

Mr Cole (AKA Monkey Boy):

My little monsters...

bill.