Andrew Rutz's blog

Short Course Nationals, 2008

Well, it's over. One and a half years of planning and practice... and it's over. The 2008 USMS Short Course National Championships held at the University of Texas' Swim Center are now in the history books. (short course is a swimming term that means the pool is setup to be 25 yards in length (as opposed to a 50 meter pool, eg).

I injured myself a month before the meet, so I wasn't able to swim, but I was able to "see my name in lights":

I was head of the Registration committee, and I also worked as Head Timer on Saturday. There were 1865 USMS (Masters) swimmers from all around the country. There were all kinds of swimmers ... all the way from Olympians (eg, Josh Davis and Shaun Jordan) to "normal people" whose story might even make you cry.

I had entered the 100 yard breastroke, 100 IM, and 200 IM. The only digital record of my being there is this video of my heat in the 100 IM from floswimming.org. I use the term "video" loosely, as you will not SEE me in the video... you will only be able to hear the announcer say my name. By the way, for all you swimmers and swim-fans out there, floswimming.org is an amazing website that hosts the coolest swimming videos.

I can honestly say that I was bummed when the meet ended. I had so much fun being in that atmosphere.. and contributing to what many said was a wonderful meet. Thank you, Whitney, for helping me improve as a swimmer. 'Sorry that I could not get in that water and show my stuff :-)

Posted at 08:35PM May 10, 2008 by Andrew Rutz in Swim  |  Comments[0]

OBP debug #1

This is my first in a hopeful series of blogs regarding debugging at the OBP (OpenBoot Prom) level.

OBP is the system firmware that executes after POST (Power-On Self-Test) and before (in the normal case) Solaris. (Note that another Client of OBP is the kernel debugger, kmdb. In that case, kmdb would be loaded, and then transition control to Solaris)

---

One can either intentionally or unintentionally arrive at OBP's prompt. It is affectionately known as the "OK prompt":

ok 

To intentionally arrive at the OK prompt, one can use a command sequence such as

# sync; sync
# halt

or

# init 0

or

# shutdown -i0 -y -g0

(The above command says to go to run-level zero and give a grace period of zero seconds).

Unintentional transfer of control to the OK prompt can be caused by traps un-/mis-handled by Solaris.

---

To load the OBP commands (eg, Forth words) that are specific to Solaris debugging, one needs to tell the Solaris boot code to load the forthdebug module. One does this by adding this to the /etc/system file and rebooting:

# /etc/system
set forthdebug = 1

Setting forthdebug to be non-zero causes the misc/forthdebug module to be loaded and processed. It contains Forth code; it does not contain SPARC code. The most important step in processing the module is to read the module's contents into Solaris' heap and then have OBP have its Forth interpreter interpret the Forth code. This is done by Solaris passing the virtual address of the start of the Forth code to OBP via the Services API (see below).

---

To allow for symbolic access to Solaris addresses (from the OK prompt), one needs to load the obpsym module, which translates numeric addresses into symbolic addresses (on behalf of OBP). For the interested, this module leverages a couple of the Calls that are in the OBP API. A compliant OBP implementation implements a set of Services. Address translation is part of that Service interface.

The obpsym module can be loaded from the Solaris command line:

# modload -p misc/obpsym

Note that you want to reboot before you load the obpsym module. If you reboot after, you no longer have obpsym loaded.

---

The system is now almost prepared to debug at the OBP level. Use one of the above ways to get to the OK prompt, or use telnet to send a BREAK character from the console port:

# Ctrl-]
telnet> send break
ok

---

The last step is to execute this Forth command (eg, word)

ok kdbg-words

This enables OBP's Forth implementation to see the Forth code that was obtained from the misc/forthdebug file. One can run the following command to view the order in which OBP will lookup an entered command. The name of the left-most vocabulary has the highest priority:

ok order
context: kdbg-words forth forth options re-heads root     current: kdbg-words

Our next lesson will extend what we have learned here.

Oh... if you're wondering how to search for an OBP command/word, use this:

ok sifting thread

This will find "thread" as a sub-string in any word of any vocabulary... defined by the current vocabulary search order (eg, as shown by order).

Posted at 04:41PM Mar 03, 2008 by Andrew Rutz in obp-debug  |  Comments[1]

Woodworking Projects

Here are links to pictures of woodworking projects I have completed.

• jul08: Oak closet shelving
• jun08: Shaker firewood box
• may08: Shaker stepstool
• jan08: Serving Tray made of Cherry
• jan08: Plant Stand (in the Greene & Greene style)
  • part one
  • part two
• oct07: Stickley No. 79, a three-shelf bookcase
• aug07: Occasional Table (Black Walnut)
• aug07: Two pieces made of pine
• jul07: Country Toy and Tool Chest
• 2005: Greene & Greene picture frame

...and these projects are currently "in the works":

• Limbert wastepaper box
• Tool tote-box

...oh... and I have to mention my Saturday in Dallas at an "antique and vintage tool sale". Here's a piece of advice: you know you're at an "antique and vintage tool sale" when someone mistakes you for the proprietor and asks: "...uh, where do you have the 11-point, breasted rip saws made by Disston ?". ..and you suddenly find that whatever intelligence remains from your college degrees is slowly receding... and you answer in an authoritative voice: "uh... I think they are near the twelve-point... ...breasted... ...rip... saws... made... ...by ...Disston...." ;-)

Here's a five-minute video from the tool sale; count how many "tool geeks" you can find !

...and last, but not least, here's an album from a week-long hand-tool woodworking class that I attended at Homestead Heritage in Waco, TX.

Posted at 09:39AM Oct 29, 2007 by Andrew Rutz in :-)  |  Comments[0]

My opensolaris fixes ...

Here's a list of the code modifications I've contributed to OpenSolaris .

My largest contribution is this pseudo-driver written from "whole cloth". Imagine: ...a device-driver that actually contains comments.... who would have thought? :-)

Posted at 04:35PM Jul 26, 2007 by Andrew Rutz in Solaris  |  Comments[0]

Unbelievable coach ...

Well, I just finished two months of one-on-one lessons with one of the best swim coaches I will ever have the pleasure of learning from. I went once or twice a week for a one-on-one, one-hour lesson with Rada Owen, who swam at Auburn University and earned a gold medal at the 2000 Olympic games.

We worked on all four strokes, and I was amazed at her "eye" for technique. She's known in the "swimming world" for having that skill, but it was a privilege to witness it and be a direct beneficiary of it.

I recorded notes after each session, and will refer to them in the future during my attempts at becoming a significantly better swimmer.

I liken Swimming to Golf. I was amazed at how difficult Golf was to learn. It took me eight years to become somewhat accomplished; I've already spent seven years at swimming. Each sport requires an unbelievable level of skill and technique. Each seems to be a "sport of opposites". Eg, if you want to hit the ball high in golf, you swing the club down. If you want the ball to go to the left, you make the club-head go to the right. If you want the ball to go far, you have to swing "easy". What looks like a game dominated by hands and arms... is actually a game that lives or dies based on one's footwork. Power does not come from the appendages; it comes from the body's core.

I have less dualisms for swimming, as I'm not an accomplished swimmer... but the issue of "core power" is patently true. ...and the idea about "swimming harder to go faster" is certainly not the way to go. Eg, it takes a smooth application of power to move through the water faster. A smooth application (eg, acceleration) of power in golf leads to a significantly more controlled and faster clubhead.

I know the deceptive elusiveness of these two sports has been what's caused them to grab my attention so deeply. I greatly thank Rada for helping me understand and to learn some more of the basics and intricacies of swimming.

Posted at 10:23PM Jun 30, 2007 by Andrew Rutz in Swim  |  Comments[0]

Recursive panic

I will dispell any rumor to the contrary: it is possible to call panic() during the execution of panic(). As is true for any (language that supports it and) function that performs it, this is an "act of recursion" !

I recently stumbled across an instance of a recursive-panic in a core file I was analyzing. After attaching mdb to the core file and running ::msgbuf to dump the console messages written by the kernel to its circular, in-memory buffer, I noticed a "curious, single line" buried within its output:

panic sync timeout

An annotated replay of the "tail" of ::msgbuf's output is shown here, along with our "curious, single line":

# ls
bounds    unix.0    vmcore.0
# mdb -k 0
Loading modules: [ unix krtld ... ]
> ::msgbuf
[...]
panic[cpu17]/thread=3000372c680:
BAD TRAP: type=28 rp=2a101871200 addr=124ed4c mmu_fsr=0

event_pvr_exec-5:
integer divide zero trap:
addr=0x124ed4c
pid=3751, pc=0x124ed4c, sp=0x2a101870aa1, tstate=0x80001606, context=0x1d64
g1-g7: ffff, fc00, 4000, 0, 0, 0, 3000372c680
000002a101870f20 unix:die+9c (28, 2a101871200, 124ed4c, 0, 2a101870fe0, 10000)
  %l0-3: 0000000000000000 0000000000000028 000000000000000f 000000000179ed54
  %l4-7: 0000030000074f40 0000000000000061 000000000179ed53 0000000001074000
000002a101871000 unix:trap+560 (2a101871200, 58, 0, 0, 300019aa000, 3000372c680)
  %l0-3: 0000000000000000 0000060012e8efd0 0000000000000028 00000600133be388
  %l4-7: 0000000000000000 000006001f282700 0000000000000001 00000300019aa180
000002a101871150 unix:ktl0+48 (4000, 40, 0, 0, 3002189493c, 0)
  %l0-3: 0000000000000007 0000000000001400 0000000080001606 00000000010195c4
  %l4-7: 0000000000f0007f 0000000000000100 0000000000000000 000002a101871200
000002a1018712a0 ssd:ssd_get_physical_geometry+204 (3001eda2898, 2a10187140c, 0, 0, 0, 60003df3580)
  %l0-3: 0000030021894938 0000000000000024 0000000000000000 000000007fffffff
  %l4-7: 0000000080000000 0000000080000000 000003002189493c 0000000000004000
000002a101871350 ssd:ssd_resync_geom_caches+b4 (60003df3580, ffffffff80000000, 200, 1, 3ec1, ff)
  %l0-3: 00000000000000ff 000000000000003f 0000030023a7b420 000003000e760780
  %l4-7: 00000000003f0000 0000000000ff0000 0000060003de0b00 0000000000020a2a
000002a101871420 ssd:ssd_validate_geometry+b4 (60003df3580, 1, 3000372c838, 1, 7, fc000092)
  %l0-3: 000002a101871484 0000000000000006 0000000000000200 000000000179ed52
  %l4-7: 0000000080000000 000000000000003c 0000060003de0b00 00000000018b8c00
000002a1018714e0 ssd:ssd_ready_and_valid+2cc (60003df3580, 3000372c680, 60003df36c0, 0, 2, 2)
  %l0-3: 0000000000000000 00000000018acd68 0000030000008ff0 0000060003de0b00
  %l4-7: 0000000000000000 0000000000000000 0000060003de0b00 0000060003de0b00
000002a1018715f0 ssd:ssdopen+260 (8, 60003df3580, 3, 18cfb68, 2, 60003df3650)
  %l0-3: 0000000000000000 0000000000000002 000000760000013b 0000000000000000
  %l4-7: 0000060003de0b00 0000000000000001 00000000018c7c00 0000000000000000
000002a1018716a0 specfs:spec_open+438 (2a101871930, 1, 60002c01ee8, 18c88f0, 600138a0de8, 30010464c80)
  %l0-3: 0000000000000001 0000060016670f40 0000000000000000 00000000ffffffff
  %l4-7: 0000000000000000 00000600166b13e0 00000600138a0e70 000000760000013b
000002a101871760 genunix:fop_open+78 (2a101871930, 2, 60002c01ee8, 1, 60016670f40, 1)
  %l0-3: 0000000000000001 00000300000a4dd8 00000600068a23d8 00000600068c9738
  %l4-7: 0000000000000000 00000600068a23f0 00000600068c9738 0000000000000004
000002a101871810 genunix:vn_openat+4d4 (0, 99c, 0, 1, 0, 1)
  %l0-3: 000000007fffffff 0000000000000000 0000000000000000 0000000000000000
  %l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
000002a1018719d0 genunix:copen+260 (ffffffffffd19553, ffbff60c, 0, 3b99c, 0, 1)
  %l0-3: 00000600133be590 00000600133be5b0 000003501b56956d 0000000000000000
  %l4-7: 00000000018a7c00 000000000000099c 0000000000000012 0000060012e8efd0

syncing file systems...
 
panic[cpu17]/thread=3000372c680:
panic sync timeout

dumping to /dev/md/dsk/d1, offset 64422019072, content: kernel

The characters in green were output by the first call to panic(), which was caused by a division-by-zero trap that occurred in the function ssd_get_physical_geometry(). The output shown in red is the output from the second panic (eg, the panic that occurred while servicing the first panic).

Our "curious, single line" (1), shown below, is found at the articulation between the first and second panics. The first panic calls vfs_syncall() to write any in-memory representation of a filesystem to its associated backing-store (2). As will be seen shortly, this execution times-out, and panic() is called again (3), this time with the argument "panic sync timeout". panic()'s design for re-entrancy ensures that most of its implementation is not repeated. Progress resumes with the writing of the core file to disk (4).

[...]
syncing file systems...             (2)
 
panic[cpu17]/thread=3000372c680:    (3)
panic sync timeout                  (1)

dumping to /dev/md/dsk/d1, offset 64422019072, content: kernel    (4)

panic()'s re-entrancy includes the use of "panic triggers", which are three global booleans which are atomically updated at specific points in panic()'s control flow:

  /*
   * Triggers for panic state transitions:
   */
 int panic_quiesce;                      /* trigger for CALM    -> QUIESCE */
 int panic_sync;                         /* trigger for QUIESCE -> SYNC */
 int panic_dump;                         /* trigger for SYNC    -> DUMP */

panic_trigger() is the function that atomically updates a specified trigger and returns True if the current call to the function caused the first (and only) change from an un-triggered state to a triggered state. If panic_trigger() returns False, then panic() knows that the associated trigger has already been triggered, and that an associated critical section of panic()'s control flow must not be run (again).

The high-level control flow of panic() is:

panic
  vpanic
    panicsys
      ...
      flush_windows()       ;; (on SPARC, write register stack to memory stack
      panic_trigger(&panic_quiesce)  ;; is first time panic has been called ?
      if is-first-panic
        switch thread's stack pointer, %sp, to panic_stack
      save registers to memory stack
      panicsys
        ...
        if (on_panic_stack)     ;; eg, "if is first-panic ..."
          ...
          panic_stopcpus()      ;; idle the current thread on (n-1) cpu's
          ...
          save lots of values in panic_xxx variables   ;; eg, panic_thread
          ...
        if panic_trigger(&panic_sync)    ;; have we not sync'd filesystems ?
          set cpu's interrupt level to ten  ;; CLOCK_LEVEL
          do_polled_io = 1;                 ;; force drivers to use polling vs interrupts
          vfs_syncall()

        if panic_trigger(&panic_dump)   ;; have we not written core file ? 
          set cpu's interrupt level to ten
          do_polled_io = 1;     ;; us polling vs interrupts  
          dumpsys()
        do a reboot    ;;  mdboot(A_REBOOT, ...)

The call-sites to the three triggers are shown in bold characters. As can now be seen, the first panic triggers panic_quiesce and panic_sync, and is in its call to vfs_syncall (to sync all the filesystems). When this sync'ing causes a panic, this second panic does not (re-)trigger either panic_quiesce or panic_sync, thus skipping much of panic's control flow. However, the second panic does trigger panic_dump, causing a call to dumpsys() to create the core file (using polled-IO, vs. interrupts).

"OK, this all 'sounds nice', but how do we know that this is what happened ?"

The trick lies in knowing which stack pointer, %sp, to use when "dumping the stack".

The stack above that is shown in green is the stack at the time the current thread's division-by-zero trap forced control to flow to die()'s call to panic(). The thread that enters panic() (and carries out panic()'s actions) is the same kernel thread (0x3000372c680) that was handling the open() system call (see frame at bottom of stack).

We also know that this same thread was the thread that was executing in vfs_syncall() when it panic'd again, as the same thread address is shown for the second panic:

[...]
syncing file systems...           
 
panic[cpu17]/thread=3000372c680: <----- !!!
panic sync timeout                 

dumping to /dev/md/dsk/d1, offset 64422019072, content: kernel    (4)

We also know that panic() obviously tries hard to ensure that "the dump" shows the state as it existed for any first panic, which means anything in the green part of the stack obviously only represents the first panic.

The trick is to look in what turns out to be the most obvious place, the per-thread state (kthread_t) of the thread that caused the second panic. When a thread is idled, Solaris saves the thread's last PC and stack pointer in kthread_t.t_pcb:

# mdb ...
> 3000372c680::print kthread_t t_pcb
{
    t_pcb.val = [ 0x105bec4, 0x2a100beb2d1 ]
}
> 0x105bec4/ia
panicsys+0x48:  call      -0x1aa88      
panicsys+0x4c:  

If we dump the stack using this stack-pointer, we obtain a lot of insight:

> 3000372c680::print kthread_t t_pcb.val[1] | ::stack
vpanic+0xcc(11d8d20, 2a100bebe78, 22bc8af, 64, 1813c00, 0)
panic+0x1c(11d8d20, 18ab000, 0, 18ab000, 1, 11d8c00)
cyclic_fire+0x64(60003df4eb8, 14b5188803b50, 3b9ac800, 0, 60003dfcb08, 60003e6f6f8)
cbe_level14+0x38(0, 0, e, 300019aa000, 20080, 100b170)
current_thread+0x140(1, 6001708de98, 1, 2710, 10741b8, 183c800)
clnt_cots_kcallit+0x580(6001708de40, 64, 0, 6001708de40, 6001708de60, 6001708deb8)
rfscall+0x58c(60002d52380, 15, 18e9ab0, 3c, 1841dd0, 18371e0)
rfs3call+0x60(60002d52380, 15, 7afa466c, 1842098, 7afa4820, 1842000)
nfs3_commit+0xc0(3001052f500, 1, 30012db13d0, 60002c01ee8, 30016159a38, 300161599c8)
nfs3_dispose+0x2c0(3001052f500, 32300000, 8000, 0, 60002c01ee8, 30016159bd8)
page_release+0xf8(70007de1680, 1, 168, 1836bb8, 1836800, 18a7000)
pvn_getdirty+0x1fc(70007de1680, 400, 55, e1c0fa355, 0, 1)
pvn_vplist_dirty+0x324(0, 400, 0, 0, 10000, 3001052f500)
nfs_putpages+0xcc(3001052f500, 0, 0, 400, 60002c01ee8, 600076241c0)
nfs3_putpage+0xcc(3001052f500, 0, 0, 1, 60002c01ee8, 0)
rflush+0x1c4(30024040000, 60002c01ee8, 30014f97480, 180a9, 7048efec, 3)
nfs3_sync+0x2c(0, 7048f000, 60002c01ee8, 1, 0, 0)
vfs_sync+0x98(18ae560, 185d550, 185dad0, 0, 0, 1)
vfs_syncall+0x48(a, a, 18b3000, 11d8000, 0, 1)
panicsys+0x4e8(3000372c680, 60002c01ee8, 184b800, 1813c00, 1077800, 0)
vpanic+0xcc(10741b8, 2a101870fa8, 2000, 1, 8, 8)
panic+0x1c(10741b8, 28, 2a101871200, 124ed4c, 0, 1c)
die+0x9c(28, 2a101871200, 124ed4c, 0, 2a101870fe0, 10000)
trap+0x560(2a101871200, 58, 0, 0, 300019aa000, 3000372c680)
ktl0+0x48(4000, 40, 0, 0, 3002189493c, 0)
ssd_get_physical_geometry+0x204(3001eda2898, 2a10187140c, 0, 0, 0, 60003df3580)
ssd_resync_geom_caches+0xb4(60003df3580, ffffffff80000000, 200, 1, 3ec1, ff)
ssd_validate_geometry+0xb4(60003df3580, 1, 3000372c838, 1, 7, fc000092)
ssd_ready_and_valid+0x2cc(60003df3580, 3000372c680, 60003df36c0, 0, 2, 2)
ssdopen+0x260(8, 60003df3580, 3, 18cfb68, 2, 60003df3650)
spec_open+0x438(2a101871930, 1, 60002c01ee8, 18c88f0, 600138a0de8, 30010464c80)
fop_open+0x78(2a101871930, 2, 60002c01ee8, 1, 60016670f40, 1)
vn_openat+0x4d4(0, 99c, 0, 1, 0, 1)
copen+0x260(ffffffffffd19553, ffbff60c, 0, 3b99c, 0, 1)
syscall_trap32+0xcc(ffbff60c, 0, 3b99c, 303240ff, 80808080, 1010101)

The green part of stack is the stack we saw when we ran ::msgbuf, and it represents the canonical stack as was present when the first call to panic() was done. The red part of the stack represents where the first call to panic() is within its control-flow. We can see the call to vfs_syncall(), and we can see for this core file that NFS and RFS code are being used to write out pages to one of their filesystems.

We also see current_thread() being called, which is a well-known function within Solaris. It runs as part of executing a high-level interrupt (eg, an interrupt whole PIL (Processor Interrupt Level)) is greater than LOCK_LEVEL (which currently is ten). As an aside, a non-high-level-interrupt's handler will have intr_thread on the stack). current_thread() calls cbe_level14(), which is used in the Cyclics implementation; "level14" gives a clue that this is a handler for a high-level interrupt whose PIL is fourteen.

The blue part of the stack represents the second panic.

To prove that this stack trace generates the "panic sync timeout" message, we disassemble the code at "cyclic_fire+0x64", as this is the last stack frame before the second panic occurs:

> cyclic_fire+0x64
> cyclic_fire+0x64::dis
cyclic_fire+0x3c:               sllx      %g4, 3, %g3
cyclic_fire+0x40:               ldx       [%g3 + %i5], %i1
cyclic_fire+0x44:               cmp       %i1, %l2
cyclic_fire+0x48:               bg,pn     %xcc, +0x80   <cyclic_fire+0xc8>
cyclic_fire+0x4c:               clr       %i3
cyclic_fire+0x50:               sra       %g2, 0, %o1
cyclic_fire+0x54:               sethi     %hi(0x3b9ac800), %i2
cyclic_fire+0x58:               add       %i5, %g3, %l4
cyclic_fire+0x5c:               add       %i2, 0x200, %l3
cyclic_fire+0x60:               mov       %l1, %o0
cyclic_fire+0x64:               call      -0x114        <cyclic_expire>
cyclic_fire+0x68:               mov       %l4, %o2
[...]

We see a call to cyclic_expire(), and from the source code, we know its signature is

static void cyclic_expire(cyc_cpu_t *cpu, cyc_index_t ndx, cyclic_t *cyclic)

From the instruction at cyclic_fire+0x68, we see that register %o2 received its value from local register %l4. If we dump our blue/red/green stack using ::stackregs instead of ::stack, we'll see this stack frame for the call to cyclic_expire:

> 3000372c680::print kthread_t t_pcb.val[1] | ::stackregs
[...]
000002a100beb5f1 cyclic_fire+0x64(60003df4eb8, 14b5188803b50, 3b9ac800, 0, 60003dfcb08, 60003e6f6f8)
  %l0-%l3:      60003c8a810      60003dab8c0    14b5188803da8         3b9aca00
  %l4-%l7:      60003e6f770    14b51995f5d50                5                1
  cbe_level14+0x38:        call      +0xa8338      
[...]

Dumping the memory referenced by %l4 will show us the third argument to cyclic_expire, which is of type cyclic_t:

> 60003e6f770::print -t cyclic_t      
{
    hrtime_t cy_expire = 0x14b5188803b50
    hrtime_t cy_interval = 0x3b9aca00
    int (*)() cy_handler = deadman
    void *cy_arg = 0
    uint32_t cy_pend = 0
    uint16_t cy_flags = 0
    cyc_level_t cy_level = 0x2
}

This cyclic runs the function deadman(), which has a few purposes, one of which is to check if the sync that is done as part of panic() has timed-out. deadman() checks the panic_sync trigger to see if a sync is in progress. If a time-out has occurred, we can see that line 1833 (below) calls panic() with our "curious, single line":

  1794  static void
  1795  deadman(void)
  1796  {
  1797          if (panicstr) {
[...]
  1819
  1820                  /*
  1821                   * If we are generating a crash dump or syncing filesystems and
  1822                   * the corresponding timer is set, decrement it and re-enter
  1823                   * the panic code to abort it and advance to the next state.
  1824                   * The panic states and triggers are explained in panic.c.
  1825                   */
  1826                  if (panic_dump) {
[...]
  1831                  } else if (panic_sync) {
  1832                          if (sync_timeleft && (--sync_timeleft == 0)) {
  1833                                  panic("panic sync timeout");
  1834                                  /*NOTREACHED*/
  1835                          }
  1836                  }
[...]

So, the Big Lesson here is that every line panic() does (and does NOT) print could be important in understanding what really occurred. Read every line of panic's output !!! =:-)

Posted at 11:12PM Jun 29, 2007 by Andrew Rutz in debug  |  Comments[0]

Who needs a compiler ?

Yes, compilers are useful tools if one is going to run the created executable MANY times, but if the executable is only run a FEW times, then Interpretation is the way to go... or... if you were in the situation I was in, then binary rewriting is the best of both worlds.

I wanted to try a fix to libpicldevtree.so, but I did not have a Solaris 9 development environment at my disposal, and I did not want to wait the several hours it takes to download and build the closure of the files needed to build my lib.

My modification was to the picld binary, which runs as a daemon. I connected mdb to it and disassembled the function-of-interest. Specifically, the instruction at offset 0x28 was the one that needed to be modified:

# ps -ef | grep pi[cl]
    root   341     1  0 16:15:16 ?        0:01 /usr/lib/picl/picld
# mdb -p 341
Loading modules: [ ld.so.1 libthread.so.1 libc.so.1 libnvpair.so.1 ]
> libdevinfo_init::dis -n 20
libpicldevtree.so.1`libdevinfo_init:    save      %sp, -0x70, %sp
libpicldevtree.so.1`libdevinfo_init+4:  call      +8            <libpicldevtree.so.1`libdevinfo_init+0xc>
libpicldevtree.so.1`libdevinfo_init+8:  sethi     %hi(0x13800), %o1
libpicldevtree.so.1`libdevinfo_init+0xc:mov       %i0, %l3
libpicldevtree.so.1`libdevinfo_init+0x10:       add       %o1, 0x218, %o1
libpicldevtree.so.1`libdevinfo_init+0x14:       mov       %i1, %l2
libpicldevtree.so.1`libdevinfo_init+0x18:       add       %o1, %o7, %l1
libpicldevtree.so.1`libdevinfo_init+0x1c:       sethi     %hi(0xdc00), %o1
libpicldevtree.so.1`libdevinfo_init+0x20:       ld        [%l1 + 0x16c], %o0
libpicldevtree.so.1`libdevinfo_init+0x24:       call      +0x140c8      <PLT=libdevinfo.so.1`di_init>
libpicldevtree.so.1`libdevinfo_init+0x28:       add       %o1, 0x307, %o1
libpicldevtree.so.1`libdevinfo_init+0x2c:       orcc      %g0, %o0, %l0

I wanted to change the instruction so that 0x327 was added to %o1, not 0x307. This can easily be done to the executing process by using these mdb commands:

> libdevinfo_init+0x28/i
libpicldevtree.so.1`libdevinfo_init+0x28:       add       %o1, 0x307, %o1
> libdevinfo_init+0x28/X
libpicldevtree.so.1`libdevinfo_init+0x28:       92026307
> libdevinfo_init+0x28/W92026327
libpicldevtree.so.1`libdevinfo_init+0x28:       0x92026307      =       0x92026327
> libdevinfo_init+0x28/i
libpicldevtree.so.1`libdevinfo_init+0x28:       add       %o1, 0x327, %o1

However, picld is executed during init(1M) processing, and so the modification must be done to picld's binary file, so that the modification would be available on each of the several reboots that I was doing. (I was measuring the modification's effect on boot time, and therefore my needs were biased towards one who selects Interpretation over Compilation: e.g., I only need to run my binary several times; I was experimenting; I was not yet "married" to my modification).

As a result, I used the wonderful utility dd(1M) to modify libpicldevtree.so so that I replaced the original ADD instruction with the ADD instruction that I needed. The following script does the trick:

# cat a.sh
#!/usr/bin/ksh

in=libpicldevtree.so.1
out=${in}.new

# copy the first 4483. four-byte "blocks" from the input-file to the output-file.
#  note that the blocks are numbered from 0 to 4482.
#
dd if=$in of=$out bs=4 count=4483

# write the 32 bits of our new ADD instruction to the output-file at block 4483.:
#
echo "\0222\002\0143\047" | dd of=$out bs=4 seek=4483 count=1

# skip over the four-byte blocks (0-4483.) in both the input-file and output-file
#  and then commence copying each four-byte block from input-file to output-file until
#  end-of-file is reached.
dd if=$in of=$out bs=4 skip=4484 oseek=4484

The only remaining issue is: "where did all those magic numbers come from ?"

We begin with disassembling the libdevinfo_init function using the dis(1) command:

# dis -F libdevinfo_init libpicldevtree.so.1
                ****   DISASSEMBLER  ****

disassembly for libpicldevtree.so.1

section .text
libdevinfo_init()
        45e4:  9d e3 bf 90         save         %sp, -0x70, %sp
        45e8:  40 00 00 02         call         0x45f0
        45ec:  13 00 00 4e         sethi        %hi(0x13800), %o1
        45f0:  a6 10 00 18         mov          %i0, %l3
        45f4:  92 02 62 18         add          %o1, 0x218, %o1
        45f8:  a4 10 00 19         mov          %i1, %l2
        45fc:  a2 02 40 0f         add          %o1, %o7, %l1
        4600:  13 00 00 37         sethi        %hi(0xdc00), %o1
        4604:  d0 04 61 6c         ld           [%l1 + 0x16c], %o0
        4608:  40 00 50 32         call         0x186d0
        460c:  92 02 63 07         add          %o1, 0x307, %o1
        4610:  a0 90 00 08         orcc         %g0, %o0, %l0

We can see our ADD instruction in bold at byte offset 0x460c (which is relative to the start of the binary file). Our goal is, therefore, to copy bytes 0 through (0x460c - 1), replace the four bytes at 0x460c, then copy the remaining bytes starting at (0x460c + 4) until the end-of-file.

Even though 0x460c is a hexidecimal number in units of bytes, it will be easier to think of our problem in units of 32-bit words, as this is the size of the item we are re-writing (an instruction, which is four bytes on this SPARC binary). As a result, the following command computes how many four-byte "blocks" reside before byte offset 0x460c:

# echo "460c%4=D" | mdb
                4483

This command divides the hexidecimal number 0x460c by the number of bytes in a four-byte block and then translates the value into decimal, as that is the radix that dd uses. The "prolific" use of "4483" in the above script now becomes evident: we copy blocks 0 through 4482, re-write block 4483, then copy blocks 4484 until the end-of-file.

The only "incantation" left to resolve is this line in the script:

# write the 32 bits of our new ADD instruction to the output-file at block 4483.:
#
echo "\0222\002\0143\047" | dd of=$out bs=4 seek=4483 count=1

This line writes an octal representation of our new ADD instruction's hexidecimal representation (0x92026327) to stdin. dd uses this as the value of the input-file and then writes this four-byte block to the output-file. The seek argument tells dd to move the current file pointer (in the output-file) to block index 4483; the writing commences there.

0x92026327 was shown above to be the encoding of the ADD instruction that we want. Since echo only seems to be able to write the binary representation of a single octal digit, we have to compute the octal representation of each of the four, eight-bit bytes of 0x92026327. Using mdb, one gets:

sh> mdb
> 92=O
                0222
> 2=O
                02
> 63=O
                0143
> 27=O
                047

We can check our work by issuing this command and seeing the bolded digits, which represent the instruction that we modified:

# dd if=libpicldevtree.so.1.new bs=4 skip=4483 | od -x |more
0000000 9202 6327 a090 0008 1280 0004 0100 0000
0000020 81c7 e008 91e8 2001 4000 502e 0100 0000
0000040 e204 6024 9290 0008 1280 0004 d024 6000
0000060 81c7 e008 91e8 2001 9010 0018 9210 0019
[...]

However, the greatest satisfaction would have been if the file's checksum would have been identical, yet the number-of-blocks and number-of-bytes are identical:

# sum libpicldevtree.so.1*
10840 162 libpicldevtree.so.1.new
10808 162 libpicldevtree.so.1.orig
# ls -l libpicldevtree.so.1*
-rw-r--r--   1 root     other      82692 Jun 25 15:46 libpicldevtree.so.1.new
-rwxr-xr-x   1 root     sys        82692 Jul  4  2005 libpicldevtree.so.1.orig

In the end, though, after manually stopping picld, installing my binary, and re-starting picld, the output of prtdiag was correct, which meant that picld had successfully loaded. Also, /var/adm/messages did not show any errors).

As a result, a quick script to perform some binary re-writing greatly improved the turnaround time regarding running my experiment. Have fun (and danger!) with your own binary-rewriting. One last tip: the binary encoding for a SPARC NOP instruction is 0x100.0000. ...or, in our "pig-octal", it is:

echo "\01\00\00\00"

Posted at 04:47PM Jun 25, 2007 by Andrew Rutz in debug  |  Comments[0]

"We can rebuild it..."

No, this is not a quiz to see if you can name all the parts of a handplane. :-)

This is a picture of an "exploded view" of one of my maternal grandfather's handplanes. Bill was both a professional cabinetmaker and carpenter in a Midwest city that knew both long, humid summers and the equally challenging freezing winters. My Mom tells of stories where customers would wait for a slot in my grandfather's schedule versus accepting the work of another. She also tells of assisting her Father when she was a child by scrounging a jobsite for bent or reusable nails ... putting them in a coffee can ... to be used again ... 'for it was the age of the Great Depression, and the "Greatest Generation" were still children... 'just beginning to internalize the principles founded on self-sacrifice.

...and so if I can bring this handplane back to life, maybe some of Bill's skill will still be in the handplane's patina, and I'll... 'er.... we'll be able to build something... together.

Posted at 07:22PM Jun 18, 2007 by Andrew Rutz in :-)  |  Comments[0]

The "24 Hours" have finished !

Well, here's a photo of my completed workbench. The magazine article predicted a duration of "24 hours", but it took me six months of weekends and inexperienced woodworking hands. However, I'm extremely pleased with the results.

On the front is, appropriately enough named, a front vise that consumed the majority of my time. I purchased the iron mechanism that consists of a mounting plate, guide rods, an ACME threaded screw, and two face plates. I purchased some hard maple and milled it to a thickness of 2.5 inches or so, and drilled holes so that the maple face plates could be attached to the iron face plates.

Between the pair of legs closest to the camera are a pair of stretchers. The end of each stretcher has a tenon that fits into a mortise in the leg. Each stretcher was glued and pegged to the leg using a dowel. A decorative black walnut cap was mounted in each peg-hole so that it was flush with the Southern Yellow Pine of the leg. The rear pair of legs were connected in the same fashion. These two end-assemblies were connected (along the long dimension) using a 2 x 8 stretcher that uses both mortise/tenon joinery and a brass/hardware bench bolt. The bench bolt allows the table to be disassembled... though I think this 300 pound table is probably easier to move by temporarily raising it onto casters !

I drilled a row of holes in the top for "bench dogs", along with one hole in the top of the outermost vise jaw. One can then use the vise with the bench dogs to hold an item of arbitrary length.

I finished the project by chamfering the edges of all legs and stretchers, and applying a tung oil-based finish to seal the wood's pores.

The coolest part during the making of the bench was when I realized that I would be able to do something with the bench that no one else will ever be able to do: I was able to "use the bench" before I "built the bench". (clue: once the vise is attached to the top, it becomes an immediate "friend" in the workshop...

Posted at 06:47PM Jun 18, 2007 by Andrew Rutz in :-)  |  Comments[0]

Woodworking bench...

so... I think there's something in the genes... I fell in love with woodworking in the sixth grade, but never had resources until now to do anything about it. I've made a few picture frames in the last couple years, but my largest and most current project is the building of a workbench. Actually, my ultimate need was to use some handplanes and chisels on some of my work... which meant I needed a vise to hold the workpiece... which meant I needed a thing to fasten the vise to... (...which, for the attentive reader... :-) describes a set of dependencies not too unlike that of my favorite nursery rhyme...

I've been building this "24-hour workbench" for the last six months =:-).

My bench is still in the midst of being constructed. Here is the table top, along with the vise. It's currently being supported by several lumber pallets:

...and here is the leg-assembly. There are mortise and tenon joints between all leg pieces, along with some bench bolts that join the two end-assemblies together.

My hopes are to create some workpieces which are at most even half as good as those of my maternal grandfather, who was both a Master Carpenter and Cabinetmaker.

Posted at 09:38PM May 12, 2007 by Andrew Rutz in :-)  |  Comments[0]

"Free" golf lesson...

If you want to see what an awesome golf swing looks like, you go to pgatour.com and find when the (American) PGA Tour is in a city near you, and you hope that Steve Elkington is playing that day. You bring your chair and a bottle of water, and watch him hit balls on the driving range, then you walk 18 holes with him. His swing was voted "Best Swing on Tour" for several years... even when a guy named Tiger Woods was around. I've been watching him since 1991; 'the swings don't get any better than this.

The following photo is from 2006, when he played in San Antonio:

Posted at 10:29AM May 07, 2007 by Andrew Rutz in :-)  |  Comments[0]

"Patently obvious"...

Well, the concept was "patently obvious" to the first author of this patent, and since I implemented a major part of the Idea, I was made second author.

"Ahhhh.... THIS is what it feels like to be a University professor!..." :-).

The patent is for an error-injection system that allows re-use of "C" code fragments that can be combined using a simple (as in Simple Mail Transfer Protocol =:-)) GUI-based development environment. The code fragments are combined using the semantics of a simple language that allows has-a relationships, Logical AND, and Logical OR.

Posted at 10:11AM May 07, 2007 by Andrew Rutz in :-)  |  Comments[0]

Books I've read since January, 2007 ...

• jul08:John Adams, by David McCullough
• jul08:Founding Brothers; The Revolutionary Generation, by Joseph J. Ellis
• jun08:Alexander Hamilton, by Ron Chernow (730 pages!)
• may08:Finding Jefferson; ... the First Amendment in an Age of Terrorism, by Alan Dershowitz
• may08:China Road; A Journey into the Future of a Rising Power, by Rob Gifford
• may08:April 4, 1968; MLK Jr's Death and How it Changed America, by Michael Eric Dyson
• may08:Appointment in Samarra, by John O'Hara
• apr08:American Pastoral, by Philip Roth
• apr08:Bad Samaritans; The Myth of Free Trade..., by Ha-Joon Chang
• mar08:Standing at Armageddon, by Nell Irvin Painter
• mar08:Day of Empire, by Amy Chua
• mar08:The Trial, by Franz Kafka
• feb08:For Whom The Bell Tolls, by Ernest Hemingway
• feb08:The Conscience of a Liberal, by Paul Krugman
• feb08:Uh-Oh, by Robert Fulghum
• feb08:It Was on Fire When I Lay Down on It, by Robert Fulghum
• feb08:Maybe (Maybe Not), by Robert Fulghum
• feb08:From Beginning to End -- The Rituals of Our Lives,by Robert Fulghum
• feb08:True Love, by Robert Fulghum
• feb08:To a God Unknown, by John Steinbeck
• jan08:As I Lay Dying, by William Faulkner
• jan08:All I Really Needed to Know I Learned in Kindergarten, by Robert Fulghum
• jan08:God is not Great, by Christopher Hitchens

• dec07:The Wind-up Bird Chronicles, by Haruki Murakami
• dec07:What on Earth Have I Done ?, by Robert Fulghum
• dec07:Write It When I'm Gone, by Thomas M. DeFrank
• nov07:The Second Civil War, by Ronald Brownstein
• nov07:Second Chance; Three Presidents and the Crisis of American Superpower, by Zbigniew Brzezinski
• oct07:Hackers and Painters: Big Ideas from the Computer Age, by Paul Graham
• sep07:An Unfinished Life: John F. Kennedy, 1917-1963, by Robert Dallek
• sep07:Che Guevara, A Revolutionary Life, by Jon Lee Anderson
• sep07: Joseph McCarthy, The Misuse of Political Power, by Daniel Cohen
• aug07: Imperial Life in the Emerald City, by Rajiv Chandrasekaran
• aug07:Death of a Salesman, by Arthur Miller
• aug07:The Seekers, by Daniel J. Boorstein
• aug07:A Woman in Charge, by Carl Bernstein
• aug07: Martin Eden, by Jack London
• jul07: Oil on the Brain, by Lisa Margonelli
• jul07: Milton Friedman, a Biography, by Lanny Ebenstein
• jun07: Seventeen Traditions, by Ralph Nader
• jun07: The Battle for God, by Karen Armstrong
• jun07: Jerusalem: One City, Three Faiths, by Karen Armstrong
• jun07: The Autobiography of Malcolm X, as told to Alex Haley
• may07: Basic Judaism, by Milton Steinberg
• may07: Dreams from My Father, by Barack Obama
• may07: The Long Tail, by Chris Anderson
• may07: Reading Judas: The Gospel of Judas and the Shaping of Christianity, by Pagels and King
• may07: The Handplane Book, by Garrett Hack
• apr07: House of Saud, by Said K. Aburish
• apr07: Religious Literacy: What Every American Needs to Know -- and Doesn't, by Stephen Prothero
• apr07: Audacity of Hope, by Barack Obama
• feb07: Freakonomics, by Stephen Levitt
• feb07: The Iranians: Persia, Islam and the Soul of a Nation by Sandra Mackey
• feb07: The Reckoning: Iraq and the Legacy of Saddam Hussein by Sandra Mackey
• feb07: Palestine: Peace, not Apartheid, by Jimmy Carter
• jan07: Lebanon: Death of a Nation by Sandra Mackey
• jan07: Passion and Politics: The Turbulent World of the Arabs by Sandra Mackey
• jan07: American Gospel: God, the Founding Fathers, and the Making of a Nation by Jon Meacham
• jan07: The Gifts of the Jews: How a Tribe of Desert Nomads Changed the Way Everyone Thinks and Feels, by Thomas Cahill

Posted at 12:36PM Apr 11, 2007 by Andrew Rutz in :-)  |  Comments[0]

A swimming-first ...

...for someone who in terms of Time came in last. :-)

Well, I had an awesome time at the "Zones meet", even if my times weren't that fast.... but... hey... I did something I never ever thought would be on one of the stops in my Life's journey: I swam in an organized swim meet.

A swim meet is not just about the Swimming; it's about the people you meet... and I want to say I enjoyed spending time with Brad, and Bob, and Mary, and Sally, and Ed, and Henry, and Ande, and Jon, and Keelah... as well as our awesome coach, Whitney.

I also want to thank Ande Rasmussen, who took time out of his busy meet-schedule to show me some fundamentals on breaststroke-starts. Ande swam NCAA Division 1 at UT-Austin, is a polymath ;-), and has a website called swimfasterfaster.com. Thanks, Ande!

..and now... on to the ugly details... :-)

The above represents the first "organized swim" of my life, a 100yd individual medley (25yds of butterstruggle.... er, "butterfly"... 25yds of backstroke, 25 of breaststroke, and 25 of freestyle). I've worked a bunch of meets, so I kinda knew how it was all supposed to play out, so I had the "macro" thing going okay... it was just the details that all seem to get cloudy when one is "under the gun".

Time either slowed down, or there actually was a long pause due to some reason... but I had longer than I thought I'd have before I actually stepped onto the "block". It was kind of like what they say the transition of Death ;-) is like... that you see images flash by. Well, I played some of them back: my Mom sitting at the high school pool for what I think was three summers in a row... and me unable to decipher the biomechanical permutation of swimming... yet my Mom always encouraging me that I could do it... that I would "get it"... I thought of the summers in Laguna Beach... where I was afraid to take the "boogie board" past the first set of waves... because I didn't think I'd make it back... of how I never liked "swim parties" as a kid... cuz I "didn't know what to do".... I fast-forwarded to 1982-83.. when I road my bike two-hundred miles a week and ran 30-40 miles... how I'd watched the first televised Ironman competition from Hawaii... and how I'd thought that ONLY IF I knew how to swim... that some day I could make it there... and then there was 1984... and a new-found reason X:-) to learn how to swim... how I took a swim-class at UC Irvine and completed a mile as our final-exam... how I spent some lunches at Xerox in El Segundo swimming in a 40yd pool... and barely making it more than two laps without being exhausted.... how I swam on my own in Austin... five days a week... for two years...

..and then the sound that I'd only ever heard said for others: "Swimmers! take your mark!". ...and then the gun... and then an almost literal loss of sound when I broke the water's plane. I'd never had that feeling before. It was like I was in a dream... or at least what TV tries to portray as a dream ... or a "water world"... where someone is cut off from all sound. It was eery.. yet sublime. My dive had probably been too "steep", and so I had a ways to come up to "hit air"... but I actually swam butterfly in public... if only for 15 yards or so. I then touched with both hands... commenced the backstroke leg... yet fail to find anything to say about it here... as I don't remember anything about it :-). I can't say that I even remember anything distinctly about the breaststroke or "free" legs either, except that the waterline seemed much more turbulent than I'd expected.. and so I could not compare my position to others... which.. naturally... I used to brainwash myself into thinking I was near the lead in my heat.. and that.. therefore... many of the people were yelling for ME! :-) "...ah... Ignorance is bliss". Well, as can be seen from the above, I was the slowest in my age-group (eleventh place). I swam the butterfly and backstroke in a combined 47.24 seconds, and I needed 51.31 seconds to complete 25yds of breaststroke and 25yds of freestyle. ...but... I tell you... when I touched the wall... in my mind... I was out-touching Michael Phelps for "gold" at the Olympics.

It took me 47 years to swim those 100yds; the next 50yds were only an hour away...

...technically, they were one hour, forty-four seconds, and ninety-seven hundredths of a second away... ;-)...

What's NOT shown above is that for this heat I happened to be seeded in lane four, the position where the highest seed (within the heat) is seeded. I moved onto the block in lane four at the sound of the referee's first buzzer, and feelings of Irony replaced the feelings of Nostalgia that were generated by my first race. Here, the "slowpoke" of my age-group... ...by the "random walk" of Life is handed a "lane-four card"... and allowed the privilege to... ...at least until the "gun" went off... to look like he knew what he was doing.

I'm being hard on myself, but this race was fun, even though I tore a flap of skin off the "big toe" of my back foot when I pushed off from the blocks. I again dove too deep, and Ande later told me I have lots of room for improvement on my "underwater work" (eg, how far I travel under water after leaving each wall). I finished last in my heat and last in my age-group... but first in my personal pantheon of accomplishments ;-) A while after this heat, Ande worked with me for at least fifteen minutes on how to improve my underwater work. He looked so darn efficient when he was demonstrating what to do. There's a story passing around the UT swim-locker-room that Ande can swim to the bottom of the eighteen-foot-deep diving well and hold his breath for 15 minutes; I tried to go down to about twelve feet, and my ears felt like they were "giving up the ghost".

My last planned race was to be the 200 breaststroke:

I got to race next to a TXLA teammate, Robert Hughes, and he was giving me tips almost all the way out onto the bulkhead (the movable, spanning structure that straddles the entire width of the pool and that supports the weight of the blocks, timers, athelets, and officials). I was in lane eight, so I could only look to the left (eg, lane seven) to see any form of human life (eg, to see where I stood in relation to at least ONE competitor). Robert already had me by half a pool-length by the time Robert touched for his 50. I felt OK through the first 100, and then it hit me as to why they call it a "200"... because one has to swim 100 MORE yards!!! :-) My turns became even more ragged; I was too close to the water surface when pushing off the wall. Ande's assistance was useful, but it in no way had become "second nature" so soon.

However, the "2" in the right-most column in the above set of results means I earned TWO POINTS!!!! for the team! These are my first, official points ever won by myself in an organized meet!. Ok, Ok... I know... "settle down"... but... heh... I know I won them because the field was so small (eg, there weren't even eight competitors (in my age-group) for the 200-breast)... but... heh... "points is POINTS" ! :-)

...and, I have to apologize to Ed Coates for this, but I swam my last race using Ed's name. It was the 200-free-relay, and I wasn't scheduled to participate, but Ed's poor intra-day health led him to go home early... and allowed me to swim a relay. My relay-mates, Brad, Henry, and Robert, concluded I should lead off the relay, which would remove the chance of me disqualifying ("DQ'ing") the team due to mis-timing the "hand-off" between relay-mates. I led off with a "stellar" ;-) 37.54 for the 50-free. My goggles loosened as I broke the water-plane, and so I could not easily see the oncoming wall. It appeared to be close.. then far away.. then close... ...and I just thought of how stupid (and painful) it would be to misjudge the wall. I'd either run into it and crack my skull... or I'd do a flip-turn and having nothing to push off of. The touch-pads on each end of pool were neon-yellow, so there was no excuse regarding "visual target capability"... it's just that there was too much water inside my goggles. Luckily (and maybe due to the thousands of times I've done flip-turns after 25 strokes (uh... I mean... "yards") ;-) ... I guess one has a feel of where the wall should be. My water-angel was looking out for me. In any event, it REALLY is more fun swimming on a relay team; there's an amplification of the notion of Team; one wants to do well for the others.. not for one's self.

Well, that was my day at the University of Houston... at my first organized swim meet. I played the wrong sports in high school; my high school "do-over" would contain: "swimming, basketball, and golf" ... or "cross-country, swimming, and basketball"... or "swimming, basketball, and volleyball"... hmm... I guess I'm not going to solve that "do-over" now... so... I'll just focus on what I can do: and that's trying to improve at swimming.... NOW!

Oh yeah... here are the relay results:

I guess a "38-second fifty" was good enough to help TXLA score ten points! yowsa!

Posted at 07:20PM Mar 25, 2007 by Andrew Rutz in Swim  |  Comments[0]

What's wrong with this picture ?

Okay... this is assumedly the last time you'll ever see myself and who is currently deemed to be the greatest all-around swimmer in the world (Michael Phelps) on the same swim-deck at the same time.

I worked as a timer at the American Short Course Championships a couple years ago, and I most likely polluted :-) the intent of someone's picture by being in the line-of-sight with Michael :-)

Michael is in middle of pic, with one foot on track-stand, goggles on, and blue cap with the "C" (of "CW", for "Club Wolverine") showing. I am over his left shoulder, recording the time of the swimmer in my lane (Brendan Hansen, World record-holder in 100m and 200m breaststroke).

Posted at 12:17PM Mar 22, 2007 by Andrew Rutz in Swim  |  Comments[1]