mjnelson's 'blog Live From the Cliffs of Insanity

My friend Tim Haley has a handwritten sign in his office. It says, quite simply, "Don't assume." I don't know the original context, but since Tim was part of the core release staff for Solaris 9, I reckon it had something to do with the whole "innocent until proven guilty" concept that the dwindling minority still embraces.

But I'll twist it around, and tackle an assumption near and dear to my own heart right now. From the first time we learn to program in a high level language, most of us go through the "edit, compile, test" cycle, often ad nauseum. And every time the "test" step fails, we go all the way back to "edit," because the problem is always in the code we wrote.

That's wrong, of course. Compilers are programs, too, and sometimes they've got mistakes. Late in the Solaris 10 development cycle, I got drafted to help out one of the Solaris Volume Manager project teams investigate one of those dreaded, logically impossible bugs. (I was part of the SVM team, but hadn't been working on this particular project.) Basically, along with some other output, the metarecover(1M) command was spitting this out:

The following soft partitions were found and will be added to
your metadevice configuration.
 Name            Size     No. of Extents
    d8202          204800         1

...which is all well and good, except for the fact that the value d8202 really was impossible. Because SVM keeps the 13-bit metadevice number and the 5-bit set number encoded in an 18-bit minor number. And, as any calculator-savvy high school student can confirm, (8202 > 8191).

But maybe we were just printing out the whole value, and not masking off the set number. That would explain things quite nicely, and the fix would be easy. On to the code:

	/* display configuration updates */
	(void) printf(dgettext(TEXT_DOMAIN,
	    "The following soft partitions were found and will be added to\n"
	    "your metadevice configuration.\n"));
	(void) printf("%5s %15s %18s\n",
	    dgettext(TEXT_DOMAIN, "Name"),
	    dgettext(TEXT_DOMAIN, "Size"),
	    dgettext(TEXT_DOMAIN, "No. of Extents"));
	for (i = 0; i < num_sps; i++) {
		(void) printf("%5s%lu %15llu %9d\n", "d",
		    MD_MIN2UNIT(MD_SID(un_array[i])),
		    un_array[i]->un_length, un_array[i]->un_numexts);
	}

Dang! I was wrong. There it is, clear as day. The MD_MIN2UNIT macro did the masking:

	#define NBITSMINOR32 18
	#define MD_BITSSET 5
	#define MD_SET_SHIFT (NBITSMINOR32 - MD_BITSSET)
	#define MD_MAXUNITS (1 << MD_SET_SHIFT)
	#define MD_UNIT_MASK (MD_MAXUNITS - 1)
	#define MD_MIN2UNIT(m) ((m) & MD_UNIT_MASK)

Good science fiction readers will recognize this scenario. Two contradictory facts, with no real middle ground. So we use Occam's Razor to pare away the alternatives, until only one remains. It will seem improbable, until viewed in the context of the invalid assumptions we have made. In this case, our two facts are the code and the output.

Did the code that we compiled really match the snippet above? Go to the bug report, find out when the error was reported. Verify that it can be reproduced reliably. Check the history of the source code. Sure enough, it happens every time, but only on one biweekly build, and the source code in question never changed.

We already verified the second fact, the output, in the previous paragraph. So what's left in between? Only the compiled code. Just to refresh our memory, we started with

		(void) printf("%5s%lu %15llu %9d\n", "d",
		    MD_MIN2UNIT(MD_SID(un_array[i])),
		    un_array[i]->un_length, un_array[i]->un_numexts);

...which, after preprocessing, correctly yields

		(void) printf("%5s%lu %15llu %9d\n", "d",
		     ( ( ( ( un_array [ i ] ) -> c . un_self_id ) ) & ( ( 1 << ( 18 - 5 ) ) - 1 ) ),
		    un_array[i]->un_length, un_array[i]->un_numexts);

But, when we disassemble the code from a system evidencing this failure:

libmeta.so.1`meta_sp_recover_from_wm+0x640:     sethi     %hi(0x3000), %l7
libmeta.so.1`meta_sp_recover_from_wm+0x644:     add       %l7, 0x2c0, %i5
libmeta.so.1`meta_sp_recover_from_wm+0x648:     sethi     %hi(0x3000), %l4
libmeta.so.1`meta_sp_recover_from_wm+0x64c:     ld        [%l3 + %i5], %i4
libmeta.so.1`meta_sp_recover_from_wm+0x650:     add       %l4, 0x2bc, %l5
libmeta.so.1`meta_sp_recover_from_wm+0x654:     clr       %i2
libmeta.so.1`meta_sp_recover_from_wm+0x658:     ld        [%sp + 0x7c], %i5
libmeta.so.1`meta_sp_recover_from_wm+0x65c:     ld        [%l3 + %l5], %l5
libmeta.so.1`meta_sp_recover_from_wm+0x660:     ld        [%i5], %o5
libmeta.so.1`meta_sp_recover_from_wm+0x664:     ld        [%o5 + 0x18], %o2
libmeta.so.1`meta_sp_recover_from_wm+0x668:     mov       %l5, %o0
libmeta.so.1`meta_sp_recover_from_wm+0x66c:     add       %i2, 1, %i2
libmeta.so.1`meta_sp_recover_from_wm+0x670:     ld        [%o5 + 0x78], %o3
libmeta.so.1`meta_sp_recover_from_wm+0x674:     add       %i5, 4, %i5
libmeta.so.1`meta_sp_recover_from_wm+0x678:     ld        [%o5 + 0x7c], %o4
libmeta.so.1`meta_sp_recover_from_wm+0x67c:     ld        [%o5 + 0x74], %o5
libmeta.so.1`meta_sp_recover_from_wm+0x680:     
call      +0x40644      

The interesting lines are 0x664, 0x670, and 0x678, which load the non-local-string printf() arguments into %o2, %o3, and %o4. They treat un_self_id, un_length, and un_numexts identically, with none of the masking specified by the MD_MIN2UNIT() macro (and shown to be happening correctly in the preprocessor expansion above).

Finally, from a stock on10 system that does NOT fail similarly:

libmeta.so.1`meta_sp_recover_from_wm+0x508:     sethi     %hi(0x2c00), %o3
libmeta.so.1`meta_sp_recover_from_wm+0x50c:     ld        [%sp + 0x7c], %l1
libmeta.so.1`meta_sp_recover_from_wm+0x510:     sethi     %hi(0x2c00), %o2
libmeta.so.1`meta_sp_recover_from_wm+0x514:     add       %o3, 0x70, %o0
libmeta.so.1`meta_sp_recover_from_wm+0x518:     ld        [%i2 + %o0], %i5
libmeta.so.1`meta_sp_recover_from_wm+0x51c:     add       %o2, 0x6c, %o1
libmeta.so.1`meta_sp_recover_from_wm+0x520:     sethi     %hi(0x1c00), %g1
libmeta.so.1`meta_sp_recover_from_wm+0x524:     ld        [%i2 + %o1], %l3
libmeta.so.1`meta_sp_recover_from_wm+0x528:     clr       %l0
libmeta.so.1`meta_sp_recover_from_wm+0x52c:     add       %g1, 0x3ff, %l2
libmeta.so.1`meta_sp_recover_from_wm+0x530:     ld        [%l1], %i0
libmeta.so.1`meta_sp_recover_from_wm+0x534:     ld        [%i0 + 0x18], %l7
libmeta.so.1`meta_sp_recover_from_wm+0x538:     mov       %l3, %o0
libmeta.so.1`meta_sp_recover_from_wm+0x53c:     add       %l0, 1, %l0
libmeta.so.1`meta_sp_recover_from_wm+0x540:     ld        [%i0 + 0x78], %o3
libmeta.so.1`meta_sp_recover_from_wm+0x544:     add       %l1, 4, %l1
libmeta.so.1`meta_sp_recover_from_wm+0x548:     ld        [%i0 + 0x74], %o5
libmeta.so.1`meta_sp_recover_from_wm+0x54c:     and       %l7, %l2, %o2
libmeta.so.1`meta_sp_recover_from_wm+0x550:     ld        [%i0 + 0x7c], %o4
libmeta.so.1`meta_sp_recover_from_wm+0x554:     
call      +0x3967c        

It's doing the masking we would expect; 0x1c00 + 0x3ff into %l2, un_self_id into %l7, masking them together into %o2.

We're almost home. We've explained the failure, but not its transient nature. In this case, further investigation revealed use of a transitional compiler during this build. How did I figure that out? The build logs generated by the nightly build script explicitly output the version information for all of the compilers. A cross check of the before, during, and after logs confirmed the transition.

I admit that this is a trivial example of debugging, and that I glossed over some of the details of the compiler switch (in fact, I left out information about which compiler was to blame, and whether this was sparc or x86...) I really wanted to emphasize two points. First: don't assume. Don't jump to conclusions about what's right and wrong, or your mind will be closed, and you'll misinterpret evidence. Second: keep detailed records. I was able to access tons of useful information during this investigation: the binary products of every build, the compiler history, the bug report detailing the symptoms and the environment, the version control log of the code, etc. Take away any part of that, and the difficulty of solving the problem increases dramatically.

Introducing the Core Release Staff

If you hang around the OpenSolaris world for long enough, you're bound to run into us. So here's a quick intro.

In his document describing the Open Solaris Development Process, John Beck talks briefly about the C-Team. That document defines C-Team as "Consolidation Team, consisting of the core release staff (technical lead, gate-keepers, etc.) responsible for managing the source base associated with a consolidation for a given release."

That's where I fit in: I'm the Tech Lead for the Operating System and Network Consolidation for what will be the next release of Solaris. Along with the rest of the C-Team and the CRT, I'm tasked with maintaining the health of the consolidation. For five of us, the core release staff, that's a full time job.

So, without further ado, let me introduce the five members of the core release staff:

• I'm Mark Nelson. Or, to be less ambiguous, I should say Mark J. Nelson. That's how my friend (and former second-level manager) Claire Giordano addresses and introduces me, with an emphasis on the "J" that always makes me smile. Like I mentioned above, I'm the Tech Lead.

  It's tempting to use Dave Platt's cat-herding analogy, but here at Sun, it doesn't really hold water. The engineers here are smart, responsible, and hard-working, and that makes my job much easier. I spend most of my time working with project teams that are getting ready to integrate their changes. With the help of the C-Team, I help them review their changes, their test plans, their test results, their process completion, and their general crossing of "t's" and dotting of "i's." Like other members of the Change Review Team (CRT), I also do the same (without the C-Team) for smaller changes and projects that aren't big enough to warrant a full review.

  Before I got into the Tech Lead business, I worked on the Solaris Volume Manager. I work in Broomfield, CO. Before coming to Sun five years ago, I worked for Ball Aerospace, where I wrote part of the control software for two of the scientific instruments on board the Spitzer Space Telescope. Does it take cool pictures, or what?!

  

  When I'm not online, I spend much of my time rock climbing and recreating in the Colorado mountains. I've been climbing for thirteen years now, and in the last few years have taken up aid climbing. I've been slowly ticking off some of the big walls in Zion Canyon, but eventually I'll make it further west and get up El Capitan in Yosemite.

• Kelli Dupee is our Program Manager. She's our interface to the Solaris Product Team, the other Consolidation Teams, and to the various engineers and project teams. She schedules and runs meetings, mediates conflicts, facilitates discussions, and keeps us on track and honest. There's a huge amount of information that flows in and out of our team, and she helps us organize and keep track of it all. Without her, it would be much more difficult for us to get the information we need from others, and vice versa.

  Kelli came to the core team from the fast-paced world of Solaris Update Releases, where she did similar work, with a smaller team, on a shorter release cycle. She works in Menlo Park, CA.

  Curiously, Kelli's real life resembles work more than she might care to admit. She helps her husband and son juggle school, Cub Scouts, work, play, and life in general. I still haven't figured out when she finds time to sleep...

• Peter Dennis is my Assistant Tech Lead, and also the Triage Lead. He's one of the main reasons that my job is so easy. Generally, by the time I'm ready to respond to a problem, he's already dealt with it effectively, and taken steps to prevent something similar from going wrong in the future. More specifically, Pete closely monitors incoming bugs, regular regression test results, and generally outstanding issues, and drives resolution as appropriate. Somehow, he also finds time to pick up just about all of the random, stray tasks that would otherwise fall between the cracks.

  Pete's background at Sun makes him ideally suitable for this position, too: he's an engineer in the Solaris Sustaining organization. That makes him a veteran firefighter, quick decision maker, fast processor of information, bug triager, and all-around multitasker. He works in Guillemont Park in the UK, which gets us some nice added round-the-clock coverage.

  When Pete's not out battling the Demons of Stupidity, he still multitasks pretty dramatically. In addition to being the father of two inexplicably attractive, delightful children, he's also an avid gamer. Really avid; he and his team travel and compete internationally.

• Danek Duvall is our Gatekeeper. He's responsible for keeping the source code (the gate) stable and available 24x7. That starts with care and feeding of the storage and gate machine hardware, and system administration of the gate and build machines. But it also includes constant vigilance that the developers' contributions build cleanly in all supported configurations. It's a tall order, but since he's got the power to backout or undo a change, people tend to be pretty responsive to his questions or requests for help.

  Before joining the core team, Danek worked on printing, and on the x86 port of the Java Desktop System. He was also, and still is, active on the Layered Software Architecture Review Committee. Like Kelli, Danek works in Menlo Park, CA.

  During those rare times when Danek isn't working, he's an avid amateur photographer and road biker. In fact, he and Valerie Bubb just rode in Bike4Breath, and raised a whole ton of money for the American Lung Association. Valerie refers to both Danek and me as long-haired hippie freaks; we're pretty sure it's a term of endearment.

  The last time I visited California, Danek and I got some bad advice and went to see Oldboy. It was, shall we say, disturbing. At best.

• Last, but not first, we've got Natalya Reznik. She's the newest member of our team, with the somewhat misleading title of Assistant Gatekeeper. In reality, Natalya's job is the least well-defined of the whole lot. Consider her the pinch hitter for the core team, covering for Danek as needed, and picking up odd (and hopefully interesting) tasks wherever she (or one of the rest of us) finds them.

  Natalya's previous work at Sun has been similar, in that she's always contributed to fill the most immediate needs of her group. Lately, that's been in test development for ZFS, the Solaris Volume Manager, and Solaris in general. Before coming to Sun, she worked for various companies writing software to manage storage arrays and tape backup products. Like me, Natalya works in Broomfield, CO.

  Like many outstanding graduates of the Technical University of St. Petersburg, we're pretty sure that Natalya was recruited by the KGB at a young age. Fortunately, there seems to be less work lately for Russian spies, so we get most of Natalya's time.

That's me and my team in a nutshell. I look forward to working with y'all.

--Mark

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

In what language do you plan to run OpenSolaris?

Fundamentally, it's not hard to write a localized application, right? You just list all of the different strings that your program might output, and you provide translations for them all in the language of your choice.

Since my native language is English, I would start with a file something like this:

	# usr/lib/locale/C/LC_MESSAGES/yummy.po
	msgid "I would like another beer, please."
	msgstr

After bribing enough multilingual translators, or perhaps drinking all of the beers myself and surfing on over to AltaVista BabelFish, eventually I would end up with several more files. For example:

	# usr/lib/locale/es/LC_MESSAGES/yummy.po
	msgid "I would like another beer, please."
	msgstr "Quisiera otra cerveza, por favor."

	# usr/lib/locale/fr/LC_MESSAGES/yummy.po
	msgid "I would like another beer, please."
	msgstr "Je voudrais une autre bière, svp."

	# usr/lib/locale/de/LC_MESSAGES/yummy.po
	msgid "I would like another beer, please."
	msgstr "Ich möchte ein anderes Bier, bitte."

	# usr/lib/locale/zh_HK/LC_MESSAGES/yummy.po
	msgid "I would like another beer, please."
	msgstr "我會要其它啤酒, 請。"

But sometimes, as they say, el diablo is in the details. Instead of "you" and "your program" and "the language of your choice," let's talk about "hundreds of contributors" and "hundreds of programs and libraries" and "all of the languages spoken by your customers."

Now we've got a much more interesting problem space; in the ON Consolidation command and library domains alone, a quick grep reveals over 22,000 unique messages:

	fivetwelve {4}% grep '^msgid' SUNW_OST_OS*.po | wc -l
	   22625
	fivetwelve {5}% 

As you might expect, we've developed both processes and tools to help us manage this complexity. Also as you might expect, those processes and tools are somewhat arcane. And, since they're largely integrated into the build system, they're further obfuscated by some pretty heavyweight makefiles. Just for grins, let's see if we can shed a little light into the dimly lit corner of Hell in which these makefiles reside.

If you're more interested in the bigger picture, you'll want to shout out to the folks over in the Internationalization and Localization Community. But if what you really want is some quick and dirty information on how to hook into the ON makefiles, grab a cup of coffee and dig in.

We'll start with individual commands and libraries, then describe the magic that binds it all together. In general, for any directory below usr/src, you should be able to make _msg and generate the catalogs for all of the subdirectories in that part of the source tree. So, for every intermediate directory, you'll need to be sure the following macros and rules are set in the makefile:

	MSGSUBDIRS= some list of subdirectories
        
	_msg := TARGET= _msg

	_msg: $(MSGSUBDIRS)

	$(MSGSUBDIRS): FRC
		@cd $@; pwd; $(MAKE) $(TARGET)

	FRC:

What does that do? It tells us that, whenever we try to build the _msg target in this directory, we really need to build it in each of the specified subdirectories. You can find an example of this in the ON commands makefile.

At some point, you'll decide that "all the messages below this point belong together, and should be collected into a single catalog." When you decide that, you'll want to specify the following in your makefile:

	POFILE= the name of your single message catalog

	$(POFILE): list of sources
		build rule

	_msg: $(MSGDOMAINPOFILE)

	include $(SRC)/Makefile.msg.targ

The included file, usr/src/Makefile.msg.targ, contains the rules that will actually take your message catalog and put it into a staging area, where it will later be combined with other catalogs from the same text domain. That's what the $(MSGDOMAINPOFILE) dependency does. You've got some other decisions to make for "list of sources" and "build rule," and this is where it gets harder to follow. Hang on; the next couple of examples will further clarify.

Your list of sources will contain some combination of shell scripts, lex code, yacc code, c code, or other message catalogs. To limit the scope of this lengthening diatribe, let's assume that you don't have any lex or yacc or shell scripts. Instead, you have two subdirectories (call 'em fu and bar, of course) with c code. For your list of sources, you're going to have two message catalogs, one per subdirectory:

	MSGSUBDIRS= fu bar

	POFILES= $(MSGSUBDIRS:%=%/%.po)

	POFILE= fubar.po

	_msg := TARGET= _msg

	$(POFILE): $(POFILES)
		$(BUILDPO.pofiles)

	$(MSGSUBDIRS): FRC
		@cd $@; pwd; $(MAKE) $(TARGET)

	_msg: $(MSGSUBDIRS) .WAIT $(MSGDOMAINPOFILE)

	FRC:

	include $(SRC)/Makefile.msg.targ

If you're a makefile geek, you can safely skip this explanation. Here's what you've just done: you've told make(1S) that you want to build a single catalog, called fubar.po, by combining the two catalogs fu/fu.po and bar/bar.po using the $(BUILDPO.pofiles) macro. To keep make(1S) from saying "I don't know how to make fu/fu.po," you've told it to wait until it's done with make _msg in fu before you try to reference fu/fu.po (and similarly for bar/bar.po). You could still shoot yourself in the foot by trying to make fubar.po, but that's why we limit ourselves to make _msg for this stuff. You can find an example of this in usr/src/cmd/lvm/Makefile.

Now let's finish with the trees in this example, so we can at least touch on the forest. Let's say that, in fu, you've got x.c, y.c, and z.c. Furthermore, if you run z.c through the C preprocessor, the message strings change. Here's the pertinent part of the fu makefile:

	MSGSRC1= x.c y.c
	MSGSRC2= z.c
	MSGFILES= $(MSGSRC1) $(MSGSRC2:%.c=%.i)

	POFILE= fu.po

	$(POFILE): $(MSGFILES)
		$(BUILDPO.msgfiles)

	_msg: $(POFILE)

	include $(SRC)/Makefile.msg.targ

What's different? First, you're not referencing the $(MSGDOMAINPOFILE) dependency. That's because you don't want fu.po to be treated as an independent message catalog; instead, it will be further processed into ../fubar.po by the previous makefile. Second, you're causing z.c to be preprocessed according to the .c.i suffix rule . Third, you're using the $(BUILDPO.msgfiles) macro instead of $(BUILDPO.pofiles). It's more efficient to call xgettext(1) just one time on three source files than it is to call it three times and then postprocess the resulting catalogs. And, like you might infer from the previous example, you can find an example of this in usr/src/cmd/lvm/util/Makefile.

So, with these examples, perusal of the make(1S) manpage, usr/src/Makefile.msg.targ, and usr/src/Makefile.master, and sufficient caffeine, you can probably figure out how to build your message catalogs. Take a deep breath, because we're about to shift our focus.

That takes care of your side of things; now what happens with those catalogs after you're done? If you were really paying attention, you probably noticed the following in usr/src/Makefile.msg.targ:

	$(MSGDOMAIN):
		$(INS.dir)

	$(MSGDOMAINPOFILE): $(MSGDOMAIN) $(POFILE)
		$(RM) $@; $(CP) $(POFILE) $@

and the following in usr/src/Makefile.master:

	#
	# For source message catalogue
	#
	.SUFFIXES: $(SUFFIXES) .i .po
	MSGROOT= $(ROOT)/catalog
	MSGDOMAIN= $(MSGROOT)/$(TEXT_DOMAIN)
	MSGDOMAINPOFILE = $(MSGDOMAIN)/$(POFILE)

What's going on now? We're taking all of the catalogs with a dependency on $(MSGDOMAINPOFILE) and we're copying them into a staging area. More specifically, that staging area lives in $(ROOT)/catalog. If you're building the fubar command message catalog on an x86 box, for example, that means you've just installed your catalog in proto/root_i386/catalog/SUNW_OST_OSCMD/fubar.po. And there it will stay, until make _msg gets invoked from usr/src/pkgdefs/SUNW0on.

Which brings us to usr/src/pkgdefs/SUNW0on/Makefile, which is the bridge between the tools and processes that we mentioned way back at the top of this entry. In this makefile, we find the following comment describing the _msg target:

	# The _msg target gathers the output of the top-level _msg
	# target into text-domain-specific message files under the
	# ROOTMSGDIR for packaging.

...and that's it. This takes everything from that staging area and uses it to build a package called SUNW0on. If you've read my other opening entry, then you know that "on" is short for "operating system and network code," and that we're just one of many consolidations that make up the Solaris product. All of the different consolidations send their SUNW0* packages to a group of folks that localize them and provide additional locales for Solaris.

For convenience, here's a collection of makefiles that should satisfy curiosity about the standard way to do this. These files might make useful cut and paste sources, but be careful: the messaging-related macros, rules, and includes are scattered throughout the files, and the placement (particularly of the include) is important.

• usr/src/cmd/lvm/Makefile

  This is an example of a command- or library-level makefile. It passes the message target on to its subdirectories, then combines the resulting catalogs. It then installs the combined catalog into the staging area.

  Note that it uses catalog as the target, instead of _msg. These two target values are often used interchangeably. The important thing isn't which one you choose, but that your parent and child makefiles agree. No matter what you use in your own hierarchy, your top level must handle the _msg target, because that's what it's going to get from the ON makefiles. Supposedly, _msg was used instead of msg because the message catalog extraction was retrofitted into the makefiles, and there was concern about conflicting with existing usage.

• usr/src/cmd/lvm/metassist/Makefile
• usr/src/cmd/lvm/metassist/Makefile.targ

  Together, these are an example of an intermediate-level makefile. It passes the _msg target on to its subdirectories, then combines the resulting catalogs. It does not install this catalog, but relies on its parent makefile to take care of that.

• usr/src/cmd/lvm/metassist/common/Makefile

  This is short and sweet. It uses $(MSGFILES) and $(BUILDPO.msgfiles) to create a message catalog from preprocessed C source code. It does not install this catalog, but relies on its parent makefile to take care of that.

If you're still reading, I'm amazed, but I can suggest some related questions:

• How do you provide a translation note in your catalog?
• When do you use gettext(3C) vs dgettext(3C) in your code?
• How are non-LC_MESSAGE category messages handled?
• What are some pitfalls of translating format strings?
• Why should you use the BUILDPO.* macros as the action in your own rule instead of just depending on the pofile_* targets?
• How come usr/src/cmd/chown/Makefile causes chown.po to be created without referencing any of this complicated crap?
• How do I test my internationalization?

...if somebody's interested, I can follow up on these questions or any others. Or not; perhaps I'll choose something less useful and more interesting for my next ramble through the woods.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

I said I would write more about work and climbing, but first, this random digression.

I live and work in Colorado, but the Sun engineering community is distributed globally. A large number of my coworkers live in California's San Francisco Bay Area, and work in Menlo Park. So it's remarkably useful for me to visit the Menlo Park campus once in a while. It helps me build and maintain effective working relationships.

Of course, when I visit, folks are usually working feverishly, and sometimes it doesn't take much to trigger the universal human need to blow off steam. In this case, the trigger was a friend's blog entry. I couldn't let Skrocki have all the fun, could I?

Why I should understudy Chewbacca for any upcoming Star Wars productions

• I'm already experienced: I've been cast in the titular (but non-speaking) role in the upcoming spectacular Space Monster on Danger Beach, brought to you this August by The Conviviality Community Theater & Entertainment Company.
• If I let my hair down, throw my head back, and bellow, I kind of look like a Wookie.

Ahhh. That felt good.

Howdy! I'm Mark Nelson, and I'm the Tech Lead for the next release of Solaris, and an avid rock climber.

Many of my work friends (and most of my family, for that matter) think I'm crazy because I climb. In particular, they don't understand why anybody would voluntarily spend the night on a portaledge suspended hundreds of feet above the ground.

My rock climbing friends, on the other hand, think I'm crazy because I sought out a position at work that was guaranteed to require long hours and high availability. They don't understand why anybody would voluntarily give up climbing time to sit in front of a computer.

I've decided they're all wrong. Or maybe they're all right. Or perhaps it really doesn't matter, because I love my job and I love my dangerous hobby.

Check in once a while over the course of the release; I'll be writing more about both.

--Mark