Tuesday April 21, 2009

It's all about scale...

One of the things that amazes me every time I come to China is scale. Being from the US, anything older than 250 years is ancient. Here, 250 years is just a bump on the timeline. The same sense of amazement applies to business scale. China has just over 1.3 billion people. That is a huge addressable market for manufacturers and retailers. People here seem to make their purchasing decisions on a more "functional" basis rather than the western trendy / "in thing" type of choices. For example, in the US, how many people (not counting geeks who really understand) buy iPhones or Blackberries because they see a celebrity or colleague with one? Probably more than you would think on first glance. The brand name or model name is driving alot of sales.

In China, consumers seem attracted to the functionality and utility of a given product. I don't see as many iPhones per capita as I do in the US, but they do tend to buy mobile phones that fit their needs. Browser, IM, social networking utilities, and easy text entry. Of course, glitzy and quirky are still huge in this market as well.

If we look at auto manufacturing in China (since the Shanghai auto show is this week), there are over 100 manufacturers of cars in China. Some are licensed or partnered manufacturers like Volkswagen, Jeep, and Buick, but there are still plenty of opportunities for the "little guys" like Chery and BYD. There are some seriously cool cars here in addition to the Skoda, Jeep, GM, Audi, and Volkswagens that look like home. In the US, we refer to "the big three" automakers, and heaven forbid that the economy knocks one of them out of existence. Here, if one folds because of the economy and lack of sales, there will still be 99+ others waiting to take over the market share. They make some very ugly and utilitarian vehicles here, but they also make some real stunners at incredible prices.

China's largest mobile carrier is China Mobile with over 475 million subscribers. Yes, that is 475 million. More than the population of the United States. Their operating revenue for 2008 was 412 billion RMB, or about $60 billion USD. That is huge. With those numbers, you would think that they are the only game in town. No way, there are other mobile carriers in China as well, doing (what we in the US would consider) big business.

That is just the beginning. If you want to be amazed, read up on PetroChina, Bank of China, Huawei, CCB, SinoPec, Baoshan, Cosco, the University systems here, and poke around some of the english language Chinese newspapers. The changes here over the past 15 years or so are dramatic, but the people are still as friendly and warm as ever. No where else in the world have I walked across the street from Gucci, Prada, and Rolex to eat a phenomenal $10 USD lunch. Best part of working in our Beijing office is that there are a pair of Starbucks within a block of my hotel and office to keep me caffeinated while I adjust to the timezones.

bill.

Monday April 20, 2009

Good morning Beijing.

I arrived in Beijing Sunday, just in time to watch the Shanghai F1 Grand Prix on CCTV in Mandarin. Once I saw the weather, I was glad that I didn't fly in early to go to the race. I love attending races, and five F1 races are hosted in my region, but any race that requires "monsoon tires" or "wets" just won't be on my top list of places to be. I'd make an exception for Monaco or Spa Francorchamps, of course. Throw in the likelihood of getting "general admission" tickets rather than seats under some kind of cover, and I was glad to watch from my nice, warm, dry hotel room 1000km away.

This trip is focused on a few existing (and highly valued) customers. It is what I call a "go deeper" trip. I'll be working for 3-5 days with each customer to help talk through some of their pain points and work with them to improve their operations and IT infrastructure. The issues are often "spur of the moment", or based on recent activities and drama, but digging through these as symptoms of underlying systemic issues often uncovers more interesting challenges. For instance, "that system is a piece of junk!" often translates into "No change control or configuration management, and everyone having the root password is probably not a good thing.".

I did get a chance to celebrate yesterday's news with a dinner at the Hard Rock in Beijing. Don't get me wrong, eating native is one of the perks of my job, and I love to explore cuisine in far reaching pieces of the globe (except for eel, or any animal that I would consider... ummm... gross), but once in a while you just have to get a cheeseburger and fries. The Hard Rock in Beijing is great, and the swag in the gift shop is cheaper than any other HRC that I have visited (over 30 and counting). I was bummed that they didn't have any decent hats, but I did find a couple goodies for the family at home.

The highlight of the night was the band. Didn't catch their name, but they were form Singapore and Beijing, definitely a mixed bag of nationalities. They did some nice Cure, Journey, POD, and Green Day covers with three members swapping the lead singer duties. Laugh of the night was provided by the over-animated Chinese lead singer who lost a couple buttons on his '501's during an especially spirited portion of the show. Kudos though, he never missed a note.

More later as the week gets rolling...

bill.

Friday April 03, 2009

A long week, this might help...

World, I'd like to introduce you to Goliath and Doogie. This should add a smile or two on a Friday. Goliath is a boxer/pit bull mix, a rescue dog that we were going to "keep until we found him a good home". That was 5 years ago. Doogie is a 140 lb (63.5 kg to my non-US friends) pile of pure stupid, formed into the shape of a yellow labrador retriever. Impossible you say? Labs are smart? This one took two years of daily "gotta chase the cows next door" running into the electric fence to learn (1) exactly *what* an electric fence is, and (2) running into it is a painful thing. Enjoy.

bill.

When good marketing goes bad...

Yeah. Goes bad... Like cottage cheese sitting in the back corner of your refrigerator for about a year until the power goes off for a few days while you are on vacation in the middle of summer. That bad.

I was looking through my drawer this morning for an old t-shirt to wear while welding some damage on my tractor's mower deck. Yeah, I live way out in the boondocks with a large yard for the kids to play in, and my tractor broke last fall.

I grab a random shirt from the pile, throw it on and head downstairs. All of a sudden, the message on the shirt hits me. It is an old Sun Education (AKA Sun Learning Services) shirt that says ".com your career". In 2000, that message was very cool, along with the "we are the dot in .com" on the sleeve. Unfortunately, the meaning of .com'ing has changed a bit. Back in the day, it meant taking your business online, exposing yourself to this new Internet connected world, and making Google money.

Oops. With the bursting of the .com bubble, to "dot com" something tends to mean something different these days. "Dot comming" your career would likely mean blowing it up, concentrating on fleeting opportunities, or other negative emotional responses. Especially if you lost some money on those high tech stocks in the past 8 years or so.

Oh well, the shirt keeps my winter-pale body covered, and I don't really mind if I get some dirt and crud on this one.

bill.

Wednesday February 25, 2009

A Mile High, and Enjoying Mexico...

We are in Mexico City this week for account reviews, customer visits, and other fun activities to make customers happy and revenue flow. The teams here are much more "mature", as Mexico has been a strong country for Sun and Sun PS for many years. The feeling of "teamwork" here is amazing, and the people work well together within Sun, as well as across the partners and with the customers.

Enough of the work stuff though, Mexico City is alot of fun, even if the gringo doesn't habla very much espanol. I speak border spanglish and Dora spanish thanks to my children watching Dora the Explorer and Go Diego Go non-stop. Above the Dora vocabulary, I know enough working spanglish to order beer, find a bathroom, and get home in a taxi.

A local cantina that provided a tasty lunch and liquid refreshment Monday. I have never seen a smashed up, 20 year old pickup truck with Jaguar "starfish" wheels before.

The food and drink have been great, and the local teams are very happy to share their experiences, both good and bad. I am definitely looking forward to working with the teams here on some projects in the near future. Huge bonus if that work requires that I be here, in the 80 degree sunny weather rather than home in the freezing cold and snow.

Here is something you don't see every day. Strolling downstairs to find an exit after the upstairs entrances were locked for the day, I stumbled upon this little gem. The first picture (bad picture, I know) is fairly easy to recognize, an ATM. The second picture puts this in context. The ATM is in the middle of a cube-farm inside the sun office. Safety for those in need during off-hours, and very convenient any time.

Oh well, home (finally) on Friday. Can't wait to sleep in my own bed, eat dinner with my kids, and get back to a somewhat normal life for a few weeks.

bill.

Friday February 20, 2009

Sao Paulo, Rio, and beyond...

The Emerging Markets PS team spent the past week in Brazil. A whirlwind tour of the key accounts and key opportunities in Sao Paulo and Rio. One week is definitely not enough time to spend in Brazil when you are in the office 12 hrs a day and hopping flights within the country to get from office to office.

Sao Paulo is the biggest little city I have ever seen. For a city of ~18M people, it feels like you are in a small neighborhood most of the time. There are no real "skyscraper" districts that make you feel closed in (like in New York), and no frantic hustle and bustle like Beijing or Los Angeles. The people are friendly, the account teams are eager and lively, and the work is exciting. What a cool town.

We also spent about 23 hours in Rio. Yeah. Rio definitely can not be consumed in less than one day. The atmosphere of the local office was upbeat, and the opportunities were exciting with lots of momentum. The city is amazing. The scenery (mostly from a cab) is gorgeous, with the mountains and hills, the ocean and bay, and 1930's and 1940's Spanish/Portuguese architecture. The view from the hotel at 5:30 AM as we were heading back to the airport was breathtaking, with the early morning water traffic and beach in the foreground and the hills and cityscape in the background.

Off to Mexico City Sunday. They have alot to live up to if they want to compare favorably with my Brazil experience. Down side is that Carnivale starts this weekend, and I will miss most of it flying out Sunday. I'll upload some pics when I get a chance to shuffle them off of my Fuze.

bill.

Monday December 22, 2008

Back from Beijing (and Shanghai)...

I just returned from a week long trip to our Beijing and Shanghai sales offices. I hadn't visited China since about 2001, and much has definitely changed. I suppose I should rewind a few weeks though for context.

Sun's new office in Beijing:

My blahg has been pretty much off-topic for a few months, as I have been transitioning into a new role. I am now the lead for Systems and Storage Professional Services for "Emerging Markets Region". EMR consists of China, Russia and the rest of CIS, India, Latin America, much of the middle east, and any country without a Sun office. Well, it seems that way. Tons of opportunities, tons of great people, tons of great customers.

Beijing's famous hub of "retail negotiations", the Silk Market:

This trip was awesome! The teams in Beijing and Shanghai are definitely on a roll. Lots of complex identity management work, portal integrations, and storage management projects underway. Huge thanks to Dowson for hosting us in both cities, Andy for helping to set everything up, Winston (and Jimmy via mobile phone) for helping with our social activities, Hammer, Jimmy, Arthur, and all the delivery architects for your time and for sharing the information.

Next comes Moscow and Brazil. For now, time to spend the holidays with family and friends. More to come later...

bill.

Friday November 07, 2008

SunOS Rises Again, Better than Ever!

After 15 months of bozos whacking away with power tools...

YAY!! And yes, my 2008 Hybrid Mercury Mariner has "Solaris" license plates. And yes, that is a 1975 Bricklin SV1 Gullwing in the background, and no, I haven't touched it in 3 years, and yes, if you'd like to take it off my hands, I would entertain offers.

bill.

Back to the real world...

I spent the past couple of months working on a project that had way too many lawyers involved. I didn't want to blog about it, as dealing with lawyers and "content review" for everything I decided to write during the project would have made my head explode.

Now that the project is finished, and the intellectual property sharks are done reviewing and blessing things, I feel a bit more open about sharing my experiences. I did get to work with a a great geek who also happens to be an actor. I learned a ton about storage and cloud-like things using virtualization layers.

Since the project finished, I have been working with the xVM Server folks on the Early Access Program, details here and here.

I'll throw some screen shots and info up soon. For now, I have a borrowed, single CPU, dual core, 8GB memory x2200 M2, sitting in a datacenter 1500 miles away to run my tests on. As of today, it is running the xVM Server software (EA2), with Solaris 10 update 6, a Solaris Express Community Edition release (Nevada 101a), Windows Server 2003 x64 Enterprise Edition, and Windows Server 2008 x64 Datacenter Edition. All on the same machine. All managed from a single desktop filled with console windows. My Windows Server guest systems even have remote console working, with decent performance to my desktop at home over VPN.

Yay!

bill.

Friday September 19, 2008

What's in a name...

Yeah, we all get tons of SPAM in our Inbox folders. I, for one, and definitely numb to the constant offers for medical assistance with my naughty tingly parts, offers to refinance my mortgage, and work from home and make millions in my spare time. I definitely am convinced that apricots and magic herbs from the Brazilian rain forests hold the keys to stopping aging, fixing my sore and aching joints, and removing those laugh lines and wrinkles that are appearing as I age. Numerous bankers in foreign lands would love to have my help in extricating some funds orphaned in their troubled country by a western national's untimely demise with no apparent heirs. I am flypaper for all of this helpful information.

This one, however, made me laugh, and coffee squirted out of my nose. Someone should tell "scruffy", oops, Mr. William Gill that using a gmail account for the "Reply-to", and having a sender address of "scruffy@zoom-dsl.com" makes me doubt the value to me, as a consumer, of this offer from "Barclays Wealth Home". Perhaps someone should also suggest a spell checker and/or grammar checker. I think that "United Kingdom" is spelled with a capital "K".

I needed a good laugh today. Thanks Scruffy.

bill.

Friday August 22, 2008

Wikis for Dummies...

The folks at The CommonCraft Show have produced a bunch of interesting videos in a series called "in Plain English", explaining how technology (and Zombies) work. Very Cool stuff:

Enjoy...

bill.

Wednesday August 13, 2008

Tupperware comes in sets...

Continuing where I left off, the previous blahg entries addressed installation of the Solaris 8 branded container. Those pieces covered the mechanics of the container itself. One of the key architectural decisions in this process was "where do we put the stuff?". Not just mountpoints and filesystems, we already covered that, but what pieces go on local disk storage, and what pieces go on the shared SAN storage?

Since the objective is to eventually integrate into a failover scenario, we looked at two options here. Each one has benefits and can supply a capability to our final solution. In the first case, we want to fail a container over to an alternate host system. In the second case, we want to fail a container over to an alternate datacenter. Think of these two as "Business Continuity" and "Disaster Recovery".

In the Business Continuity case, the capability to do "rolling upgrades" as part of the solution would be a huge added bonus. We decided to put the zone itself on local disk storage, and the application data on the shared SAN storage. This allows us to "upgrade" a container, roll the application in, and still maintain a "fallback" configuration in case the upgrade causes problems, with minimal downtime. Accomplishing this requires two copies of the container. Application data "rollback" and "fallback" scenarios are satisfied with the shared SAN storage itself through snapshots and point in time copies.

Similar to a cluster failover pair, both zones have their own patch levels and configurations, and a shared IP address can be used for accessing application services. Only one zone can be “live” at any time as these two zones are actually copies of the same zone “system”.

To migrate the branded container to another host system, the zone must be halted, and the shared SAN storage volumes must be detached, and unmounted from the original host system:

The detach operation saves information about the container and its configuration in an XML file in the ZONEPATH (/zones/[zonename] in our configuration). This will allow the container to be created on the target system with minimal manual configuration through zonecfg.

The detached container’s filesystem can now be safely copied to the new target system. The filesystem will be backed up and then restored on to the target system. There are many utilities that can create and extract backup images, this example uses the “pax” utility. A pax archive can preserve information about the filesystem, including ACLs, permissions, creation, access, and modification times, and most types of “special files” that are persistent. Make sure that there is enough space on both the source system and the target system to hold the pax archive (/path/to/[zonename].pax in the example) as the image may be several gigabytes in size. Some warnings could be seen during the pax archive creation. Some transient special files cannot be archived, but will be re-created on the target system when the zone boots.

On the target system, the zone filesystem space must be configured and mounted, and have “700” permissions with owner root. The /zones loopback mount must also be in place, just as in the source system.

Since the zone filesystem is not on shared storage, and will remain local to the target system, the “mount at boot” option can be set to “yes”.

Storage for the applications and data should now be imported and mounted on the target system to replicate the configuration of the source system. All mountpoints, loopback filesystems, and targets of the “add fs” components of the zone must be replicated. Once the filesystems are mounted into the global zone, the zone pax archive can be extracted. Again, care must be taken to make sure that there is sufficient space on the zone filesystem for the extraction:

The filesystem of the zone is now in place, but the zone is not yet configured into the target system. The zone must be created, modified as necessary (i.e. different network adapter hardware or device naming), and “attached” to the new host system. As a sanity check, it is highly recommended that the /usr/lib/brand/solaris8/s8_p2v command is run against the new zone to make sure that the new system “accepts” the attach of a zone created elsewhere:

The “attach” command may fail with messages about patch version conflicts, as well as extra or missing patches. Even though this is a full root zone, the detach/attach functionality makes sure that the host systems are equivalent. Some patches will be missing or extra in some cases, especially where the machine types or CPU types are different (sun4u, sun4v, HBA types and models, Ethernet adapter hardware differences, etc.). It is possible to normalize all patch versions and instances across systems of different configurations and architectures, but this involves significant effort and planning, and has no real effect on the operation of the hosting systems or the hosted zones (patching software that will never run on a given machine).

Once all errors and warnings are accounted for as “accepted deltas” or resolved, a failed attach can be forced:

Zone migration can be toggled between the machines by halting the zone, detaching the zone, moving the shared SAN storage into the target system, attaching the zone and booting the zone. Once the zone has been installed, configured, and booted on both systems, there is no need to use the s8_p2v function for migration. Strictly speaking, the “detach/attach” function is not necessary since the zone itself resides locally, and is not actually migrating, but it does provide an extra layer of protection on the non-active machine to keep the halted zone from being booted while the shared storage is not active. By setting the zone state to “detached”, the zone will not boot unless the “attach” command is executed first, providing the check for the shared SAN storage configured with the “add fs” zone configuration.

Pretty simple, huh? In fact, if you look at the above diagram, it looks mysteriously like the functionality of a cluster failover. Once we modeled and tested these actions by hand, we integrated the pair of containers into Veritas Cluster Server and managed the zones through the VCS GUI. Online, offline, failover... It all just works. Very cool stuff.

bill.

Tuesday August 12, 2008

Tupperware footnote...

A couple folks have emailed me asking about my VxVM and VxFS in this physical to virtual conversion. As I have blahg'd before, separation is a good thing. In this case, we had Veritas Volume Manager and Veritas Filesystem on the source machine, and on the target machine. Volume Management and Filesystem Management should live within the Global Zone, not in local zones (or non-global zones as they are officially called). Mixing "system" activities within zones is a BadThing[tm], especially when the zone is a branded container. Trying to use Solaris 8 filesystem utilities and disk volume management utility binaries (even through the branded containers software) against a Solaris 10 system, kernel, and possibly even a hardware architecture (sun4v) unknown to a Solaris 8 operating system is a dangerous path to walk.

Definitely too much risk in there for me to even attempt to whack it into working.

We installed out Volume Management (VxVM) and Filesystem (VxFS) on the Solaris 10 target host system using the Veritas Foundation Suite (5.0 maintenance pack 1 Rolling Patch 4). All of the storage goodies were installed and configured as local objects on the Solaris 10 host system, and mounted under the /z/[zonename]_[function] pathnames as described earlier. The lofs loopback mounts and zonecfg "add fs" pieces mapped them into the places that we wanted them to be, just providing "disk space" to the Solaris 8 branded containers. We did use zonecfg "add fs" with a type of vxfs, and it worked as advertised. In the end, we decided that the VxFS pieces are a "system function" and should be mounted in the Global Zone under /z for simplicity and consistency.

Who knows, at some point we might even use ZFS instead of the VxFS in this configuration (more on that in a later blahg entry), and this allows us to keep the "zone space" filesystem and storage agnostic.

bill.

Monday August 11, 2008

Tupperware isn't the only Branded Container in town...

For this project, two Solaris 8 branded containers were installed on the test systems using "flar" images. The containers were created “unconfigured” so that IP addresses could be assigned to avoid conflict with the source systems. Instructions from the docs.sun.com Solaris 8 branded containers administration guide were used to install and configure the test containers. The docs in the administration guide were excellent, with lots of "type this" and "you should see this" guidance.

We had two test scenarios that we wanted to play with: (1) Branded containers on SAN shared storage, and (2) branded containers hosted on local storage with SAN shared storage for application data. There are advantages and disadvantages for both, and situations where each has significant value. I'll get into the details of that in a future blahg entry.

In order to separate the "storage administration" from the “zone administration” function, storage was configured with device mounts within /z, regardless of the type (SAN, NAS or internal) of storage being utilized:

The storage for the container was mounted to the /zones/[zonename]/ path using loopback filesystems (lofs). This mount was created for testing using the command line:

So that the mount could persist for reboots, an entry was added to /etc/vfstab:

Note that the “mount at boot” option is set to “no” in this example. This is to allow for zones installed on SAN shared storage volumes to migrate back and forth between the systems. Zones installed on local, internal storage will use “yes” for the “mount at boot” option. In the shared SAN storage zones, the filesystems must be mounted as part of the zone management when rebooting, or attaching a zone into a new host:

The basic Solaris 8 branded container was created with the zonecfg command on the first Solaris 10 host system. The basic Solaris 8 branded zone contains a zonepath for the zone to be installed within, and a network interface for network access:

Extra filesystems for administrative tools and applications are mounted into the branded container from the global zone mountpoints with the “add fs” command within zonecfg. The loopback filesystem (lofs) is used to allow the Solaris 10 global zone to manage devices, volume management, filesystems, and mountpoints, and “project” that filesystem space into the branded container. While it is possible to pass physical devices in to a container, it is not a recommended architecture when at all possible to avoid, especially in the case of branded containers, where device and filesystem management would be running under the brand, while the kernel being leveraged would be of a different OS release. This loopback filesystem is defined during the zone configuration with:

Because the Solaris 10 Global Zone (GZ) can run on machines that don’t support native Solaris 8, some applications and software packages could become confused about the architecture of the underlying hardware and cause problems. The Solaris 8 branded container is shielding us from the hardware platform hosting the Solaris 10 GZ, so it is recommended that we set a machine architecture of “sun4u” (Sun UltraSPARC) for the branded container so that the hardware platform is essentially hidden from the operating environment within the container. The “machine” attribute can be assigned within the zone configuration with:

Other filesystems can be added, additional network interfaces can be defined, and other system attributes can be assigned to the branded container as needed. Once the container has been configured within zonecfg, the configuration should be verified and committed to save the data.

The branded container is now configured and ready for installation of an image (flar) from a physical system. The Solaris 8 system image is created using the flarcreate command. Make sure that the source system is up to speed on patches, especially those related to patching, flash archives (flar) and the "df" patch (137749) that I mentioned in an earlier blahg entry.

The configured zone can now be installed using the flar image of the physical Solaris 8 based system:

Once the zone has been installed, it the “P2V” utility must be run to turn the physical machine configuration into a virtual machine configuration within the zone:

The Solaris 8 branded container can now be booted. There will likely be some system startup scripts, tools, and applications that will give warning messages and errors on boot up. Remember the nature of the zone / host system relationship and decide what needs to run in a zone and what functionality should remain in the host system global zone. The zone will boot “un-configured”, and ask for hostname, console terminal type, name service information, and network configurations on the first boot. As an alternative, a “sysidcfg” file can be copied into the /zones/[zonename]/root/etc/ directory before the first boot to allow the zone to auto-configure with sysid upon first boot.

That's it. Other than fixing up the system startup scripts (/etc/rc*.d and /etc/init.d), and attaching in a copy of the source system's SAN attached data, the move is done and ready to be tested. The really cool part of this is that it just works. We were expecting some application issues and possibly some "speed of light" problems, but everything just worked. Obviously some things had to be adjusted in the branded container, disabling the Veritas Volume Manager startup, and some hardware inventory scripts that used "prtconf" to collect information, but these were identified early, and several reboots sorted out the symptoms from the zone's boot messages on the zlogin console.

More details later about migration of the zone between systems, various storage configurations that we tested, and some other (hopefully) interesting thoughts.

bill.

Monday July 28, 2008

Separation is a Good Thing...

One of the interesting side effects of using containers/zones is that it provides a layer of separation for administration functions. Along the way, we have an opportunity to look at other separations that we can provide between the various IT organizations contributing to the "service" being provided by the system.

As an example, how many of us IT types here "I need the root password" from the database or application folks on a regular basis. Well, unless you have implemented RBAC (Role Based Access Control), and have trained your developers and DBAs to the point that they know what to ask for, this is probably a common request. I am alot more comfortable as an administrator giving administrative access to a zone than I am in giving it within the Global Zone. I know that if they totally roach things in the local zone, I can still edit files and move things around from the Global Zone to fix things up, and the system is not "down" or in a state that I need to boot from DVD or Jumpstart and twiddle bits.

Implementing zones gives us a chance to integrate several areas of separation, providing simpler administration. One example, I like to separate "disk/storage administration" from system/zone administration. I mount the (non-ZFS) storage under /z and then use the loopback filesystem (lofs) to make my zonepath. for example:

/z/[zonename]_root lofs mounted on /zones/[zonename] /z/[zonename]_opt_oracle zonecfg "add fs" with mountpoint=/opt/oracle /z/[zonename]_data_oracle zonecfg "add fs" with mountpoint=/data/oracle

Obviously this is even easier with ZFS, but I might get to that another day in another blahg entry. Using the resource mountpoint and the zonename in the filesystem information allows me to use grep in interesting ways with "df" and "mount" to more easily track down where things are being mounted and used.

I also try to avoid using devices within zones. This is especially important when I am using branded containers, as it creates interesting dependencies between the Solaris 8 software and the Solaris 10 kernel interfaces. One prime example, Veritas Filesystem (vxfs) and Veritas Volume Manager (vxvm). Often, when migrating a physical Solaris 8 machine into a branded container, admins will try to move administrative functions into the container and treat it as a real machine. There are huge benefits in moving "administrative functions" into the Global Zone, and leaving the application functionality within the branded container. Just say no. Manage your local storage, SAN, and volume resources from the GZ, and just present "disk space" into the local zones.

Other administrative functions that are much easier to keep in the GZ include backups, network administration, configuration management, and performance and capacity management. I find alot of effort being spent on trying to "shoehorn" these functions into a container, when logically they belong in the "system", or GZ. Separating the "system" (there is only one, and it is a shared resource) and the "application" (there can be many, and they consume the "system) has huge benefits in the ongoing administration and maintenance efforts. This is more of a "how to think" problem than a "what to think" education.

Just my opinions, your mileage may vary, objects in mirror may be closer than they appear, etc.. Feel free to comment, debate, or object in the comments below.

bill.