"The second-named motive, ambition or, in milder terms, the aiming
at recognition and consideration, lies firmly fixed in human nature.
With absence of mental stimulus of this kind, human cooperation
would be entirely impossible; the desire for the approval of one's
fellowman certainly is one of the most important binding powers of
society.  In this complex of feelings, constructive and destructive
forces lie closely together.  Desire for approval and recognition
is a healthy motive; but the desire to be acknowledged as better,
stronger or more intelligent than a fellow being or fellow scholar
easily leads to an excessively egoistic psychological adjustment,
which may become injurious for the individual and for the community."

--From Albert Einstein's Out of my later years, 
On Education

• "Now that zfs is in the gate, maybe you can mow the lawn?"
• "Now that zfs is in the gate, can you take the kids to their tennis lesson on Saturday?"
• "Now that zfs is in the gate, maybe I should get an iPOD nano?"
• ...

In the process of trying to get Solaris compiled with the Sun Studio 10, aka, Vulcan compilers, I debugged numerous problems, some of which were not obvious at the time whose bugs they were. Here is one of them:

When libc.so.1 was compiled with GCC, everything worked fine; when it was compiled with Vulcan, all multithreaded programs hung. After some more debugging, the problem seemed to be in the Vulcan compiled usr/src/lib/libc/port/threads/synch.c: if I linked all the object files with a GCC compiled synch.o, everything worked. "It must be a compiler bug!"

I was in the middle of debugging 5 other panics and hangs at the time, so I made an offer to my compiler buddies, "Beer and lunch is on me for whoever figured it out." They tried, but at the end of the day, there was still no root cause. So I took a closer look. It appeared that the hung thread was waiting for a mutex, but nobody owned the mutex, yet the thread was not woken up. I looked at synch.c, and something caught my eye: the various lock routines calling swap32. swap32 is an inline function, and GCC and Vulcan have different inline implementations. If there is a bug there, that could explain why the GCC compiled version worked but not the Vulcan compiled verison.

        .inline swap32, 0
        xchgl   (%rdi), %esi
        .end

Let's see how it can be used:

void
spin_lock_clear(mutex_t *mp)
{
        ulwp_t *self = curthread;

        mp->mutex_owner = 0;
        if (swap32(&mp->mutex_lockword, 0) & WAITERMASK) {
                (void) ___lwp_mutex_wakeup(mp);
                if (self->ul_spin_lock_wakeup != UINT_MAX)
                        self->ul_spin_lock_wakeup++;
        }
        preempt(self);
}

Ah ha! So we did the swap, but we never returned anything to the caller. In spin_lock_clear, we were checking whatever happened to be in %rax to see if there were waiters. If %rax happened to be 0, the calling thread would think that there is no waiter to wake up, leaving the poor thread waiting for the mutex looping forever!

To fix the problem, I changed swap32 to the following:

        .inline swap32, 0
        movl    %esi, %eax
        xchgl   (%rdi), %eax
        .end

So the moral of the story is that, things are not always what they seem on the surface.

        void
        foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7)
        {
        ...
        }

        pushq   %rbp
        movq    %rsp, %rbp
        subq    $0x30, %rsp                     **
        movq    %rdi, -0x8(%rbp)
        movq    %rsi, -0x10(%rbp)
        movq    %rdx, -0x18(%rbp)
        movq    %rcx, -0x20(%rbp)
        movq    %r8, -0x28(%rbp)
        movq    %r9, -0x30(%rbp)
        ...

        argc == 0: no argument saving.
        argc == 3: save 3, but reserve space for 4 to maintain stack alignment.
        argc == 7: save 6.

        DW_AT_SUN_amd64_parmdump        = 0x2224

• How does the extra argument saving affect performance?
  
  With a 20-deep small function calls stack each with 6 arguments (to cause maximum argument saving), the impact of the extra saving is 18 nanosections, around a 10% hit.

          #define FUNC(i, j) \
                  static int      \
                  func##i(int i1, int i2, int i3, int i4, int i5, int i6) \
                  {                                                       \
                          i3 = i1 + i2;                                   \
                          i4 = i2 + i3;                                   \
                          i5 = i3 + i4;                                   \
                          i6 = func##j(i1, i2, i3, i4, i5, i6);           \
                          return (i3 + i4 + i5 + i6);                     \
                  }
      

  This is on hot cache where the first store to the stack won't suffer a page fault. Since in reality functions actually do something more complicated, the actual hit should be much smaller. If it turns out the -save_args option does affect performance of your particular application, you can always turn it off in production code.
  
  
• Why was it implemented as callee-saved instead of caller-saved?
  
  
  • Smaller code size when functions are called by many callers.
  • Avoids useless argument saving when calling assembly functions.
  • Can be enabled only on the module that's being debugged.
  
  
  
• So what does the output look like?
  
  Ha, I thought you would never ask!

  
  stack pointer for thread fffffe8123debe80: fffffe80006296c0
  [ fffffe80006296c0 unix`_resume_from_idle+0xde() ]
    fffffe8000629700 unix`swtch+0x241()
    fffffe8000629730 genunix`cv_wait+0x83(ffffffff82a44ed8, ffffffff82a44ed0)
    fffffe80006297a0 ufs`ufs_check_lockfs+0x14c(ffffffff82a44e00, ffffffff82a44eb0, 80000030)
    fffffe8000629800 ufs`ufs_lockfs_begin+0x14e(ffffffff82a44e00, fffffe8000629840, 80000030)
    fffffe8000629920 ufs`ufs_readlink+0x7e(ffffffff90377300, fffffe8000629980, ffffffff832e9428)
    fffffe8000629950 genunix`fop_readlink+0x24(ffffffff90377300, fffffe8000629980, ffffffff832e9428)
    fffffe80006299d0 genunix`pn_getsymlink+0x66(ffffffff90377300, fffffe8000629b20, ffffffff832e9428)
    fffffe8000629bc0 genunix`lookuppnvp+0x3f5(fffffe8000629ca0, 0, 1, 0, fffffe8000629e10, ffffffff8c907b80)
    fffffe8000629c60 genunix`lookuppnat+0x13e(fffffe8000629ca0, 0, 1, 0, fffffe8000629e10, 0)
    fffffe8000629d40 genunix`lookupnameat+0x88(805bd38, 0, 1, 0, fffffe8000629e10 , 0)
    fffffe8000629dd0 genunix`cstatat_getvp+0x17d(ffd19553, 805bd38, 1, 1, fffffe8000629e10, fffffe8000629e18)
    fffffe8000629e60 genunix`cstatat32+0x68(ffd19553, 805bd38, 1, fcfdbef8, 0, 10
    fffffe8000629e80 genunix`stat32+0x33(805bd38, fcfdbef8)
    fffffe8000629eb0 genunix`xstat32+0x26(2, 805bd38, fcfdbef8)
    fffffe8000629f00 unix`sys_syscall32+0x1ff()
  
      

• Solaris port to AMD64 platforms, for which we won the 2005 Chairman's Award.
• Improved write performance by 80-120% on AMD64 as measured by libMicro.
• Got -save_args option implemented by Sun Studio compilers for AMD64 so that function arguments passed via register are available to the debugger (more on this later).
• Improved debugability on AMD64 in general.

• POST (Power On Self Test)
• Parts of the System Controller software (test sequencer, domain console and domain communication channel)
• The Solaris driver for communicating with the system controller
• The Solaris drivers for DR (Dynamic Reconfiguration).