Fast Reboot becomes the default operating mode on x86 in build 112.
All new features introduced by PSARC 2008/382 Fast Reboot in build 100
are now enabled by default without the -f flag.

Table of Contents
============================================================================
1. boot-config service
    1.1 fastreboot_default
    1.2 fastreboot_onpanic

2. End Users
    2.1 Reboot to default entry in GRUB menu.lst
    2.2 Reboot to the 4th entry in GRUB menu.lst
    2.3 Reboot through the BIOS, such as for netinstall
    2.4 Fast Reboot features introduced by Phase I, now without -f

3. Kernel Developers
    3.1 Force Fast Reboot
    3.2 Debugging early panics occur between mountroot and multiuser

4. Failure Conditions
    4.1 Can't process GRUB menu
    4.2 Not all Drivers have implemented quiesce(9E)
    4.3 On xVM
    4.4 Among Blacklisted platforms
    4.5 Cannot allocate enough memory under 1G
============================================================================


1. boot-config service

        svc:/system/boot-config:default

1.1 fastreboot_default

        fastreboot_default is set to true by default except on
        guests on VirtualBox, guests on VMware, and Tyan MCP55
        reference whitebox.

        To disable the Fast Reboot by default behavior,

        # svccfg -s "system/boot-config:default" \
                setprop config/fastreboot_default=false
        # svcadm refresh svc:/system/boot-config:default

1.2 fastreboot_onpanic

        # svccfg -s "system/boot-config:default" \
                setprop config/fastreboot_onpanic=false
        # svcadm refresh svc:/system/boot-config:default

        Disabling boot-config:default has no effect on panic reboot
        behavior.


2. End Users

2.1 Reboot to default entry in GRUB menu.lst

        # reboot
    or
        # init 6

    This can be done after setting the new default entry with
    "luactivate" or "beactivate".

2.2 Reboot to the 4th entry in GRUB menu.lst

        # reboot 4

2.3 Reboot through the BIOS, such as for netinstall

        # reboot -p

2.4 Fast Reboot features introduced by Phase I, now without -f

        # reboot -e zfsbe2

        # reboot -- 'rpool/ROOT/zfsroot2'

        # reboot -- '/dev/dsk/c0t0d0s3'

        # reboot -- '/platform/i86pc/sherry-kernel/amd64/unix -k'

        # reboot -- '/mnt/platform/i86pc/sherry-kernel/amd64/unix -k'

        # reboot -- '-ks'

3. Kernel Developers

3.1 Force Fast Reboot

        If there are drivers on the system without quiesce(9E)
        support, one can first try to unload the drivers using
        modunload(1M).  If the driver does not unload, and you know
        that the device it manages does not initiate DMA or generate
        interrupts, you can patch force_fastreboot to 1 in /etc/system.

        # echo "set force_fastreboot=1" >> /etc/system
        # echo "force_fastreboot/W" | mdb -kw

3.2 Debugging early panics occur between mountroot and multiuser

        The boot-config service has dependencies on multiuser.
        Developers that need to debug early panics can patch a
        global variable fastreboot_onpanic in /etc/system.

        # echo "set fastreboot_onpanic=1" >> /etc/system
        # echo "fastreboot_onpanic/W" | mdb -kw


4. Failure Conditions

4.1 Can't process GRUB menu

        The only GRUB commands we can process right now are
            default, findroot, bootfs, kernel, kernel$, module, module$
        timeout is ignored.

        If the GRUGB menu contains other commands such as root,
        chainload, you will see something like this:

        reboot: Failed to process GRUB menu entry for fast reboot.
                Invalid GRUB entry
        reboot: Falling back to regular reboot.


        Below are other GRUB related error messages that you might see:

            Invalid argument
            xVM is not supported
            Kernel file is not unix
            No such file or directory
            Kernel path is not absolute
            Failed to open kernel file
            Bootsign not found
            Unknown bootfs filesystem
            ...

4.2 On xVM
    4.2.1. On metal, attempting to reboot to xVM:

        bash-3.2# reboot 1
        reboot: Failed to process GRUB menu entry for fast reboot.
                xVM is not supported
        reboot: Falling back to regular reboot.

4.3 Not all Drivers have implemented quiesce(9E)

4.4 Among Blacklisted platforms

    As of integration the following platforms have been blacklisted:


        1. Guests on Virtualbox.
        2. Guests on VMware
        3. Tyan MCP55 reference whitebox

    You can attempt to fast reboot by invoking reboot with "reboot -f",
    but your mileage might vary.

4.5 Cannot allocate enough memory under 1G

    WARNING: Fastboot: Couldn't allocate 0x11000 bytes below 1G to do fast
    reboot

#!/usr/sbin/dtrace -Fs

ddi_regs_map_setup:entry,ddi_dma_alloc_handle:entry
{
        this->devi = (struct dev_info *)arg0;
        @a[stack()] = count();
        @b[stringof(`devnamesp[this->devi->devi_major].dn_name),
            probefunc] = count();
}

ddi_dma_mem_alloc:entry
{
        this->devi = (struct dev_info *)
            (((ddi_dma_impl_t *)arg0)->dmai_rdip);

        @a[stack()] = count();
        @b[stringof(`devnamesp[this->devi->devi_major].dn_name),
            probefunc] = count();
}


END
{
        printa(@a);
        printa("%20s: %30s %10@d\n", @b);
        /* printa("0x%x %10@d\n", @b); */
}

clovertown-ds-1# reboot -f
Oct  1 09:31:14 clovertown-ds-1 reboot: initiated by root on /dev/console
Oct  1 09:31:16 clovertown-ds-1 rpcbind: rpcbind terminating on signal.
Oct  1 09:31:16 clovertown-ds-1 syslogd: going down on signal 15
Oct  1 09:31:16 /usr/lib/snmp/snmpdx: received signal 15
Fast reboot.
syncing file systems... done
SunOS Release 5.11 Version onnv-gate:2008-09-30 64-bit
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
DEBUG enabled
NOTICE: enabling probe 0 (::ddi_regs_map_setup:entry)
NOTICE: enabling probe 1 (::ddi_dma_alloc_handle:entry)
NOTICE: enabling probe 2 (::ddi_dma_mem_alloc:entry)
NOTICE: enabling probe 3 (:::END)
NOTICE: enabling probe 4 (dtrace:::ERROR)
Hostname: clovertown-ds-1
NIS domain name is mpklab.sfbay.sun.com
/dev/rdsk/c1d0s7 is clean
Reading ZFS config: done.

clovertown-ds-1 console login: root
Password: 

clovertown-ds-1# dtrace -ae > /var/tmp/dma.out


clovertown-ds-1# tail -30 /var/tmp/dma.out

               i8042:             ddi_regs_map_setup          1
              kb8042:             ddi_regs_map_setup          1
           mouse8042:             ddi_regs_map_setup          1
                 npe:             ddi_regs_map_setup          1
              nvidia:             ddi_regs_map_setup          1
             pci-ide:             ddi_regs_map_setup          2
                 asy:             ddi_regs_map_setup          3
                 ata:              ddi_dma_mem_alloc          3
                ehci:           ddi_dma_alloc_handle          4
                ehci:              ddi_dma_mem_alloc          4
             pci_pci:             ddi_regs_map_setup          5
             vgatext:             ddi_regs_map_setup          7
                 ata:             ddi_regs_map_setup          9
                uhci:           ddi_dma_alloc_handle         12
                uhci:              ddi_dma_mem_alloc         12
                ehci:             ddi_regs_map_setup         15
        intel_nb5000:             ddi_regs_map_setup         21
              e1000g:             ddi_regs_map_setup         32
                uhci:             ddi_regs_map_setup         56
            pcie_pci:             ddi_regs_map_setup         65
              e1000g:              ddi_dma_mem_alloc       8450
              e1000g:           ddi_dma_alloc_handle      10754
                 ata:           ddi_dma_alloc_handle      10978

x86 Fast Reboot will be available in snv_100

Table of Contents

============================================================================
1. Usage Model
    1.1. Dry run
    1.2. Fast reboot without arguments
    1.3. Fast reboot with unix arguments
    1.4. Fast reboot with root disk or pool arguments
    1.5. Fast reboot with -e option
    1.6. Fast reboot with alternate mounted root file system

2. Known-to-work Drivers
    2.1 NIC Drivers
    2.2 HBA Drivers
    2.3 USB Drivers

3. Known-to-work Platforms
    3.1 Sun Fire Servers
    3.2 Sun Blade Servers
    3.3 Desktops
    3.4 Other

4. How To Test Your quiesce(9E)

5. Known Issues
    5.1 ON bugs you might run into
    5.2 Possible conditions where fast reboot would fail
        5.2.1 Not all drivers have implemented quiesce(9E)
        5.2.2 Not enough memory
        5.2.3 Not supported

6. Features (Not Bugs)
    6.1 Transient menu entry
    6.2 Limited functionality for "uadmin 2 8"

7. A little secret

============================================================================


1. Usage Model

    1.1. Dry run: this is an undocumented internal command to check
         whether the fast reboot attempt will likely to succeed.

        # reboot -f dryrun

        On success:

            # reboot -f dryrun
            reboot: all drivers have implemented quiesce(9E)

        On failure:

            # reboot -f dryrun
            (Drivers without quiesce() implementation will be listed)
             genunix: WARNING: nvidia has no quiesce()
             reboot: not all drivers have implemented quiesce(9E)

            If you don't have console access, you can get the list of
            drivers without quiesce(9E) support like this:

            # grep "no quiesce" /var/adm/messages

    1.2. Fast reboot without arguments

        # reboot -f
        # uadmin 2 8

        NOTE: Whenever "uadmin" is used, you must make sure the boot
        archives are up to date, or behavior is undefined.

    1.3. Fast reboot with unix arguments

        # reboot -f -- '/platform/i86pc/kernel/amd64/unix'
        # reboot -f -- '/platform/i86pc/kernel/unix'

        # reboot -f -- '-k'
        # reboot -f -- '-kv'

        # reboot -f -- '/platform/i86pc/kernel/amd64/unix -k'
        # reboot -f -- '/platform/i86pc/kernel/unix -k'

        # reboot -f -- '/platform/i86pc/my-kernel/amd64/unix -k'
        # reboot -f -- '/platform/i86pc/my-kernel/unix -k'


    1.4. Fast reboot with root disk or pool arguments

        # reboot -f -- '/dev/dsk/ctd0s0'
        # reboot -f -- '/dev/dsk/ctd0s0 /platform/i86pc/kernel/amd64/unix'
        # reboot -f -- '/dev/dsk/ctd0s0 /platform/i86pc/kernel/amd64/unix -k'

       Or with root pool:

        # reboot -f -- 'rpool/ROOT/zfsbe1'
        # reboot -f -- 'rpool/ROOT/zfsbe2 /platform/i86pc/kernel/amd64/unix'
        # reboot -f -- 'rpool/ROOT/zfsbe3 /platform/i86pc/kernel/amd64/unix -k'

    1.5. Fast reboot with -e option

        # reboot -f -e s3
        # reboot -f -e s3 -- '/platform/i86pc/kernel/amd64/unix'
        # reboot -f -e s3 -- '-k'
        # reboot -f -e s3 -- '/platform/i86pc/kernel/amd64/unix -k'

    1.6. Fast reboot with alternate mounted root file system

        # reboot -f -- '/mnt/platform/i86pc/kernel/amd64/unix'
        # reboot -f -- '/mnt/platform/i86pc/kernel/amd64/unix -k'


2. Known-to-work Drivers

    2.1 NIC Drivers

        e1000g
        bge
        nge
        xge
        igb
        ixgbe
        rge
        chxge

    2.2 HBA Drivers

        nv_sata
        marvell88sx
        mpt
        aac

    2.3 USB Drivers

        ehci
        ohci
        uhci

    2.4 Misc Drivers

        bscv
        fdc
        mscsi
        ioat
        nvidia (will be in build 100)

3. Known-to-work Platforms

    Fast reboot works on the following platforms when they use devices
    with known-to-work drivers.

    3.1 Sun Fire Servers
        x2100, x2200 and their M2 counterparts
        x4100, x4200, x4500, x4600 and their M2 counterparts
        v20z, v40z
        v65x

    3.2 Sun Blade Servers
        x6220, x6250
        x8400

    3.3 Desktops
        Ultra20, Ultra40 and their M2 counterparts.
        Ultra24

        NOTE: nvidia currently does not have quiesce(9E)
        implementation.  If nvidia is the only driver shown by "reboot
        -f dryrun" as not having a quiesce(), user can safely force
        fast reboot by putting

                set force_fastreboot = 1

        in /etc/system, or

                # echo "force_fastreboot/W 1" | mdb -kw

    3.4 Other

        Intel Clovertown-based whitebox


4. How To Test Your quiesce(9E)

    - Install build 100 or above
    - bfu to the latest bits
    - Run the following command:
        # reboot -f dryrun
        # grep "no quiesce" /var/adm/messages
      Hopefully your driver is the only one without quiesce().  If not,
      file bugs against drivers without quiesce().
    - Install your driver, or install your kernel with the new driver.
    - Run the following command
        # reboot -f

      Between "rebooting..." to the Solaris should take less than 5
      seconds.  If the system hangs or goes through BIOS, the
      implmenetation is no good.

    - If the first "reboot -f" works
        * Boot kmdb
        * Put the following script in /etc/rc3.d/
      then type "reboot -f".  This will run fast reboot in a loop.  If
      the system evern drops into kmdb prompt or hangs or resets, the
      implementation is no good.

        #!/bin/sh
        LOOPFILE=/var/tmp/loops
        if [ -f $LOOPFILE ]; then
            loops=`cat /var/tmp/loops`;
        else
            loops=0;
        fi
        echo $loops
        loops=`expr $loops + 1`
        echo $loops > /var/tmp/loops
        sleeptime=`expr $loops % 20`
        sleep $sleeptime

        /usr/sbin/reboot -f


5. Known Issues

    5.1 ON bugs you might run into:

        6703055 assertion failed: (curcpup())->cpu_flags & 0x002
        6709808 assertion failed in bio.c during reboot
        6747441 GRUB/vdev_get_bootpath() and zpool_get_physpath() should
                be recursive

    5.2 Possible conditions where fast reboot would fail

        5.2.1 Not all drivers have implemented quiesce(9E)

        Sep 18 13:19:12 too-cool genunix: WARNING: nvidia has no quiesce()
        reboot: not all drivers have implemented quiesce(9E)

        If you NIC and display are the only drivers that don't support
        quiesce(9E), you can unplumb the interface first then force
        fast reboot.  Like this:

        # ifconfig iwh0 unplumb
        # ifconfig iwh0 inet6 unplumb (if you have configured inet6)
        # modunload -i 0
        # echo "force_fastreboot/W 1" | mdb -kw
        # reboot -f

        If you are happy with the result, ie, it works, you can put the
        following line in /etc/system:
        # echo "set force_fastreboot=1" >> /etc/system
        
        If you still need to following steps 1-3 before typing "reboot -f".

        5.2.2 Not enough memory

        If there is not enough memory below 1G (0x40000000) for
        building the page tables, or not enough free memory to load the
        new kernel and boot archive, the fast reboot attempt will fail
        with such messages and fall back to regular reboot:

        Fastboot: Couldn't allocate size below PA 1G to do fast reboot
        Fastboot: Couldn't allocate size below PA 64G to do fast reboot

        5.2.3 Not supported

        Fast reboot is currently not supported in xVM or zones.


6. Features (Not Bugs)

    6.1 Transient menu entry

        When rebooting to a different root disk or pool via mountpoint
        or boot environment, no transient menu entry will be added to
        menu.lst.

    6.2 Limited functionality for "uadmin 2 8"

        Most of the options available for "reboot -f" is not present
        for "uadmin 2 8".  This is a feature, not a bug.  We want to
        discourage the use of uadmin(1M) as the reboot command due to
        two main limitations:

            - uadmin(1M) doesn't update the boot archive
            - uadmin(1M) doesn't update the menu

7. A little secret

    If you know that you system is capable of fast reboot, you can do
    the following to make fast reboot the default behavior of
    reboot(1M).

            # touch /etc/fastreboot

"The second-named motive, ambition or, in milder terms, the aiming
at recognition and consideration, lies firmly fixed in human nature.
With absence of mental stimulus of this kind, human cooperation
would be entirely impossible; the desire for the approval of one's
fellowman certainly is one of the most important binding powers of
society.  In this complex of feelings, constructive and destructive
forces lie closely together.  Desire for approval and recognition
is a healthy motive; but the desire to be acknowledged as better,
stronger or more intelligent than a fellow being or fellow scholar
easily leads to an excessively egoistic psychological adjustment,
which may become injurious for the individual and for the community."

--From Albert Einstein's Out of my later years, 
On Education

• "Now that zfs is in the gate, maybe you can mow the lawn?"
• "Now that zfs is in the gate, can you take the kids to their tennis lesson on Saturday?"
• "Now that zfs is in the gate, maybe I should get an iPOD nano?"
• ...

In the process of trying to get Solaris compiled with the Sun Studio 10, aka, Vulcan compilers, I debugged numerous problems, some of which were not obvious at the time whose bugs they were. Here is one of them:

When libc.so.1 was compiled with GCC, everything worked fine; when it was compiled with Vulcan, all multithreaded programs hung. After some more debugging, the problem seemed to be in the Vulcan compiled usr/src/lib/libc/port/threads/synch.c: if I linked all the object files with a GCC compiled synch.o, everything worked. "It must be a compiler bug!"

I was in the middle of debugging 5 other panics and hangs at the time, so I made an offer to my compiler buddies, "Beer and lunch is on me for whoever figured it out." They tried, but at the end of the day, there was still no root cause. So I took a closer look. It appeared that the hung thread was waiting for a mutex, but nobody owned the mutex, yet the thread was not woken up. I looked at synch.c, and something caught my eye: the various lock routines calling swap32. swap32 is an inline function, and GCC and Vulcan have different inline implementations. If there is a bug there, that could explain why the GCC compiled version worked but not the Vulcan compiled verison.

        .inline swap32, 0
        xchgl   (%rdi), %esi
        .end

Let's see how it can be used:

void
spin_lock_clear(mutex_t *mp)
{
        ulwp_t *self = curthread;

        mp->mutex_owner = 0;
        if (swap32(&mp->mutex_lockword, 0) & WAITERMASK) {
                (void) ___lwp_mutex_wakeup(mp);
                if (self->ul_spin_lock_wakeup != UINT_MAX)
                        self->ul_spin_lock_wakeup++;
        }
        preempt(self);
}

Ah ha! So we did the swap, but we never returned anything to the caller. In spin_lock_clear, we were checking whatever happened to be in %rax to see if there were waiters. If %rax happened to be 0, the calling thread would think that there is no waiter to wake up, leaving the poor thread waiting for the mutex looping forever!

To fix the problem, I changed swap32 to the following:

        .inline swap32, 0
        movl    %esi, %eax
        xchgl   (%rdi), %eax
        .end

So the moral of the story is that, things are not always what they seem on the surface.

        void
        foo(int a1, int a2, int a3, int a4, int a5, int a6, int a7)
        {
        ...
        }

        pushq   %rbp
        movq    %rsp, %rbp
        subq    $0x30, %rsp                     **
        movq    %rdi, -0x8(%rbp)
        movq    %rsi, -0x10(%rbp)
        movq    %rdx, -0x18(%rbp)
        movq    %rcx, -0x20(%rbp)
        movq    %r8, -0x28(%rbp)
        movq    %r9, -0x30(%rbp)
        ...

        argc == 0: no argument saving.
        argc == 3: save 3, but reserve space for 4 to maintain stack alignment.
        argc == 7: save 6.

        DW_AT_SUN_amd64_parmdump        = 0x2224

• How does the extra argument saving affect performance?
  
  With a 20-deep small function calls stack each with 6 arguments (to cause maximum argument saving), the impact of the extra saving is 18 nanoseconds around a 10% hit.

          #define FUNC(i, j) \
                  static int      \
                  func##i(int i1, int i2, int i3, int i4, int i5, int i6) \
                  {                                                       \
                          i3 = i1 + i2;                                   \
                          i4 = i2 + i3;                                   \
                          i5 = i3 + i4;                                   \
                          i6 = func##j(i1, i2, i3, i4, i5, i6);           \
                          return (i3 + i4 + i5 + i6);                     \
                  }
      

  This is on hot cache where the first store to the stack won't suffer a page fault. Since in reality functions actually do something more complicated, the actual hit should be much smaller. If it turns out the -save_args option does affect performance of your particular application, you can always turn it off in production code.
  
  
• Why was it implemented as callee-saved instead of caller-saved?
  
  
  • Smaller code size when functions are called by many callers.
  • Avoids useless argument saving when calling assembly functions.
  • Can be enabled only on the module that's being debugged.
  
  
  
• So what does the output look like?
  
  Ha, I thought you would never ask!

  
  stack pointer for thread fffffe8123debe80: fffffe80006296c0
  [ fffffe80006296c0 unix`_resume_from_idle+0xde() ]
    fffffe8000629700 unix`swtch+0x241()
    fffffe8000629730 genunix`cv_wait+0x83(ffffffff82a44ed8, ffffffff82a44ed0)
    fffffe80006297a0 ufs`ufs_check_lockfs+0x14c(ffffffff82a44e00, ffffffff82a44eb0, 80000030)
    fffffe8000629800 ufs`ufs_lockfs_begin+0x14e(ffffffff82a44e00, fffffe8000629840, 80000030)
    fffffe8000629920 ufs`ufs_readlink+0x7e(ffffffff90377300, fffffe8000629980, ffffffff832e9428)
    fffffe8000629950 genunix`fop_readlink+0x24(ffffffff90377300, fffffe8000629980, ffffffff832e9428)
    fffffe80006299d0 genunix`pn_getsymlink+0x66(ffffffff90377300, fffffe8000629b20, ffffffff832e9428)
    fffffe8000629bc0 genunix`lookuppnvp+0x3f5(fffffe8000629ca0, 0, 1, 0, fffffe8000629e10, ffffffff8c907b80)
    fffffe8000629c60 genunix`lookuppnat+0x13e(fffffe8000629ca0, 0, 1, 0, fffffe8000629e10, 0)
    fffffe8000629d40 genunix`lookupnameat+0x88(805bd38, 0, 1, 0, fffffe8000629e10 , 0)
    fffffe8000629dd0 genunix`cstatat_getvp+0x17d(ffd19553, 805bd38, 1, 1, fffffe8000629e10, fffffe8000629e18)
    fffffe8000629e60 genunix`cstatat32+0x68(ffd19553, 805bd38, 1, fcfdbef8, 0, 10
    fffffe8000629e80 genunix`stat32+0x33(805bd38, fcfdbef8)
    fffffe8000629eb0 genunix`xstat32+0x26(2, 805bd38, fcfdbef8)
    fffffe8000629f00 unix`sys_syscall32+0x1ff()
  
      

• Solaris port to AMD64 platforms, for which we won the 2005 Chairman's Award.
• Improved write performance by 80-120% on AMD64 as measured by libMicro.
• Got -save_args option implemented by Sun Studio compilers for AMD64 so that function arguments passed via register are available to the debugger (more on this later).
• Improved debugability on AMD64 in general.

• POST (Power On Self Test)
• Parts of the System Controller software (test sequencer, domain console and domain communication channel)
• The Solaris driver for communicating with the system controller
• The Solaris drivers for DR (Dynamic Reconfiguration).