Kool Aid Served Daily

So I managed to really hose one of the lab machines. I entered:

# eeprom boot-file=kdmb
# reboot

Instead of:

# eeprom boot-file=kmdb
# reboot

And now the system won't boot. Furthermore, the installation failed to properly build a /boot/grub/menu.lst, so I can't get to the failsafe kernel. Both problems need to be fixed.

The first thing to do is to PXE boot the system and at the install prompt, go for single user mode:

Configuring devices.lt/init
    1.     Solaris Interactive (default)
    2.     Custom JumpStart
    3.     Solaris Interactive Text (Desktop session)
    4.     Solaris Interactive Text (Console session)
    5.     Apply driver updatesText (Desktop session)
    6.     Single user shellve Text (Console session)
    5.     Apply driver updates
Enter the number of your choice.r
Selected: 6                                        
Enter the number of your choice.r
Single user shell                                  
PROCESSING etc/user_attr
Searching for installed OS instances...
PROCESSING etc/vfstab
Solaris Nevada snv_37 X86 was found on /dev/dsk/c1d0s0.
Do you wish to have it mounted read-write on /a? [y,n,?] y
mounting /dev/dsk/c1d0s0 on /a
Solaris Nevada snv_37 X86 was found on /dev/dsk/c1d0s0.
Starting shell.have it mounted read-write on /a? [y,n,?]         ERROR: Input is required.

Notice that the console interaction sucks. Just an observation...

Anyway, simple, right:

# eeprom boot-file=kmdb
# reboot

Wrong, it still will not reboot. The eeprom settings have not changed afterwards. If we look at eeprom(1M), we find the settings are stored in /boot/solaris/bootenv.rc. So, reboot with PXE and:

# cd /boot
# cd solaris
# ls -la
total 12
drwxr-xr-x   4 root     sys          512 Apr  3 12:46 .
drwxr-xr-x   5 root     sys          512 Apr  3 12:46 ..
drwxr-xr-x   2 root     sys          512 Apr  3 12:46 bin
lrwxrwxrwx   1 root     root          38 Apr  3 12:46 bootenv.rc -> ../../tmp/root/boot/solaris/bootenv.rc
drwxr-xr-x   2 root     sys          512 Apr  3 12:46 devicedb
-rw-r--r--   1 root     sys          361 Mar 28 00:27 filelist.ramdisk
# more bo*      
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#

Okay, a safe boot from the network, so we are building a ramdisk? Yes:

# df -h
Filesystem             size   used  avail capacity  Mounted on
/ramdisk:a             130M   126M     0K   100%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   1.9G   184K   1.9G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
swap                   1.9G   324K   1.9G     1%    /tmp
/tmp/dev               1.9G   324K   1.9G     1%    /dev
fd                       0K     0K     0K     0%    /dev/fd
/dev/dsk/c1d0s0         32G   3.2G    28G    11%    /a

Where is the real file?

# cd /
# find . -name bootenv.rc
./boot/solaris/bootenv.rc
./tmp/root/boot/solaris/bootenv.rc
./a/var/sadm/pkg/SUNWrmodr/save/pspool/SUNWrmodr/reloc/boot/solaris/bootenv.rc
./a/boot/solaris/bootenv.rc
./a/bfu.child/boot/solaris/bootenv.rc
./a/bfu.conflicts/boot/solaris/bootenv.rc
./a/bfu.parent/boot/solaris/bootenv.rc
./.tmp_proto/root/boot/solaris/bootenv.rc

Oh yeah, during the boot, we told it to mount our root filesystem:

#  more ./a/boot/solaris/bootenv.rc
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#
# echo "boot-file=kmdb" >> ./a/boot/solaris/bootenv.rc

This doesn't add up, but lets also fix /boot/grub/menu.list at the same time.

# cd /a
# vi xxx
I don't know what kind of terminal you are on - all I have is 'unknown'.
[Using open mode]
"xxx" [New file] 
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris Nevada snv_30 X86
kernel /platform/i86pc/multiboot
module /platform/i86pc/boot_archive
#---------------------END BOOTADM--------------------
#---------- ADDED BY BOOTADM - DO NOT EDIT ----------
title Solaris failsafe
kernel /boot/multiboot kernel/unix -s -B console=ttya
module /boot/x86.miniroot-safe
#---------------------END BOOTADM--------------------
:wq
"xxx" [New file] 11 lines, 426 characters 
# cat xxx >> /a/boot/grub/menu.lst

The good news is that it did appear to fix the issue with grub. The bad news is:

panic: cannot open kdmb                                                  
Press any key to reboot                                      

Now I don't have to PXE boot anymore. And low and behold:

# more bootenv.rc
#
# Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# CDDL HEADER START
#
# The contents of this file are subject to the terms of the
# Common Development and Distribution License, Version 1.0 only
# (the "License").  You may not use this file except in compliance
# with the License.
#
# You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE
# or http://www.opensolaris.org/os/licensing.
# See the License for the specific language governing permissions
# and limitations under the License.
#
# When distributing Covered Code, include this CDDL HEADER in each
# file and include the License file at usr/src/OPENSOLARIS.LICENSE.
# If applicable, add the following below this CDDL HEADER, with the
# fields enclosed by brackets "[]" replaced with your own identifying
# information: Portions Copyright [yyyy] [name of copyright owner]
#

# pwd
/a/boot/solaris
# cat bootenv.rc | grep -v ^#
setprop kbd-type 'US-English'
setprop ata-dma-enabled '1'
setprop atapi-cd-dma-enabled '1'
setprop ttyb-rts-dtr-off 'false'
setprop ttyb-ignore-cd 'true'
setprop ttya-rts-dtr-off 'false'
setprop ttya-ignore-cd 'true'
setprop ttyb-mode '9600,8,n,1,-'
setprop ttya-mode '9600,8,n,1,-'
setprop lba-access-ok '1'
setprop bootpath '/pci@0,0/pci-ide@7,1/ide@0/cmdk@0,0:a'
setprop console 'ttya'
setprop boot-file 'kdmb'
boot-file=kmdb

Crap, it has to be my console over telnet! Lets fix the first form and remove the second later.

# sed 's/kdmb/kmdb/' bootenv.rc > xxx
# diff bootenv.rc xxx
41c41
< setprop boot-file 'kdmb'
---
> setprop boot-file 'kmdb'
# cp xxx bootenv.rc
# reboot

And it works!

Loading kmdb...              
SunOS Release 5.11 Version obso 64-bit                  
Copyright 1983-2006 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.ronment variables"
DEBUG enabled
Hostname: plagae 'US-English'

Now, what was I doing before I got locked out?