Paul Scott: X86

X series machines ILOM 

I have a little to write about this but I'm not sure how many people might find it useful, if you would then can you let me know?
As a quick thought, some people have asked about the Elom to Ilom upgrade process, Reading the following document http://docs.sun.com/source/820-4930-11/index.html  might help a fair bit.

let me know what you want from this (if anything).

A while ago I was asked to look at the boot_archive, It's not much but this is a start. I'll keep going with this as long as I can find something new to add. this is a bit rambling at the moment, but I will get to making it all pretty when I have the chance.

bootadm is used to both update and re-create the boot_archive file, it does this using various scripts and binarys to create a iso image (boot_archive), when grub hands over to the image (you can see the images list in in the menu.lst file) it's used to provide kernel memory modules and configuration files..

bootadm looks specifically for the following files (this list is from my machine), your list might differ slightly (no acpi etc?)

etc/rtc_config

etc/system

etc/name_to_major

etc/driver_aliases

etc/name_to_sysnum

etc/dacf.conf

etc/driver_classes

etc/path_to_inst

etc/mach

etc/devices/devid_cache

etc/devices/retire_store

etc/devices/mdi_scsi_vhci_cache

etc/devices/mdi_ib_cache

etc/cluster/nodeid

kernel

platform/i86pc/biosint

platform/i86pc/kernel

platform/i86xpv/kernel

platform/i86pc/ucode/GenuineIntel

platform/i86pc/ucode/AuthenticAMD

boot/solaris/bootenv.rc

boot/solaris/devicedb/master

boot/acpi/tables

bootadm checks a file called /boot/solaris/filelist.ramdisk you can see this list for yourself by using: bootadm list-archive.

Sometimes, when the machine reboots without you expecting it, it might well not be happy about the boot_archive, it will try to resolve any problems itself, sometimes though it can't. a fairly simple way of of replacing the boot_archive is as follows (providing the disk isn't wrecked of course)

Boot failsafe/cd/dvd (if you can, and your not really going to avoid this that I know of yet).

Mount the root filesystem on /a

cd /a/platform/i86pc

mv boot_archive boot_archive.orig

bootadm update-archive -R /a

Do try to keep the boot archive in case we can look at it and find something useful. The bootadm command might pop a few messages out bleating that it can't find some files, take a look at the list you created using bootadm - list-archive if it's not in the list then don't worry about it and wait to see what happens when you try the boot process. if it is in the list it still might not matter. for instance, on my machine it couldn't find the file "platform/i86pc/ucode/AuthenticAMD" but the machine still came up without issue. the simple message is...don't get too stressed straight away.

You can check /etc/svc/volatile/system-boot-archive:default.log for what it couldn't find, an example from my machine on a normal (none broken) boot is...

---o---

bash-3.2# more /etc/svc/volatile/system-boot-archive:default.log      

[ Sep 23 13:18:51 Enabled. ]

[ Sep 23 13:18:52 Executing start method ("/lib/svc/method/boot-archive"). ]

cannot find: /etc/devices/retire_store: No such file or directory

cannot find: /etc/devices/mdi_ib_cache: No such file or directory

cannot find: /etc/cluster/nodeid: No such file or directory

cannot find: /platform/i86pc/biosint: No such file or directory

cannot find: /platform/i86pc/ucode/AuthenticAMD: No such file or directory (<-- my machine is a authentic amd by the way)

[ Sep 23 13:18:53 Method "start" exited with status 0. ]

---o---

If you find a machine panicking (for whatever reason) on boot just as it seems to be starting, it may well be because a file is missing, specifically, if the driver_aliases file is not available or broken then you will get a mount panic. this is because the kernel cannot locate the driver that it needs to understand the underlying storage.

So, what do you do if you lose a file or two (the driver_aliases file for instance)?

boot either failsafe/cd/dvd (whichever you can) once this is done and you have managed to mount the root disk you can use

lofiadm to mount the boot archive (remember when I said it was a iso image?) as so...

lofiadm -a /a/platform/i86pc/boot_archive /dev/lofi/1

mount -F hsfs -o ro /dev/lofi/1 /mnt (the "-F hsfs" option might not work, if not then try the mount without "-F hsfs", it might work)

(as a hint/tip, you can also use this method to mount iso images and img files (filename.iso and filename.img) too)

most of these files are compressed using gzip so will need copying out and the ".gz" extension giving to them. once you have this done you can unzip them (using gzip) and you have the file you were after. once this mounts you can take whatever you need from it, bear in mind though that you might be using severely out of date files, but it might get you by and get the machine of it's knees.

If your archive is broken, see if you have a spare (backup etc), not that it's advocated, but if you ensure you have a copy either of the boot_archive or the files required you should still be ok. Just remember to associate the correct boot_archive with the lofi device for mounting. if you have mirrored disks, make sure you split the mirror before any of this just in case anything you need can be rescued from the mirror.

It's also possible that you may witness other boot failure messages, a common one is the message similar to " reading beyond end of ramdisk", this normally indicates that you have a corrupt boot_archive. check the size of the boot_archive and it's it's less than 20mb (and more likely it will be 20 bytes or so) then consider it corrupt and recreate it. Since your tinkering with the boot-archive, I'll assume your booted from cdrom (not failsafe since this is where this message can be emitted from most of the time), use bootadm update-archive -fR /<boot-archive-mountpoint> to recreate the archive. You may have to delete the boot-archive and create it again, try this before calling in the problem.

BIG NOTE:

---o---

It may be obvious, or not but; you need to know where the boot archive is when updating it. For instance, if you are booted failsafe the root of the disk with the archive your interested in will be /a or wherever you explicitly said to mount it on boot. Remember this, it might save you hours of updating archives and modifying files only to find that none of this worked for you.

---o---

Because I was asked, Here's the (simple) contents of a example bootenv.rc file (note that zfs will not have the bootpath or altbootpath lines) (/boot/solaris/bootenv.rc)

setprop ata-dma-enabled 1

setprop atapi-cd-dma-enabled 1

setprop ttyb-rts-dtr-off false

setprop ttyb-ignore-cd true

setprop ttya-rts-dtr-off false

setprop ttya-ignore-cd true

setprop ttyb-mode 9600,8,n,1,-

setprop ttya-mode 9600,8,n,1,-

setprop lba-access-ok 1

setprop prealloc-chunk-size 0x2000

setprop bootpath /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@1,0:a

setprop keyboard-layout 'US-English'

setprop console 'ttya'

Because I couldn't remember and had to scratch around for it, here's a reminder of what the contents of an automated menu.lst (/boot/grub/menu.lst)might (don't forget, things change) look like.

#---------- ADDED BY BOOTADM - DO NOT EDIT ----------

title Solaris Express Community Edition snv_107 X86

findroot (rootfs0,0,a)

kernel$ /platform/i86pc/kernel/$ISADIR/unix

module$ /platform/i86pc/$ISADIR/boot_archive

#---------------------END BOOTADM--------------------

#---------- ADDED BY BOOTADM - DO NOT EDIT ----------

title Solaris xVM

findroot (rootfs0,0,a)

kernel$ /boot/$ISADIR/xen.gz

module$ /platform/i86xpv/kernel/$ISADIR/unix /platform/i86xpv/kernel/$ISADIR/unix

module$ /platform/i86pc/$ISADIR/boot_archive

#---------------------END BOOTADM--------------------

#---------- ADDED BY BOOTADM - DO NOT EDIT ----------

title Solaris failsafe

findroot (rootfs0,0,a)

kernel$ /boot/platform/i86pc/kernel/$ISADIR/unix -s -B console=ttya

module$ /boot/$ISADIR/x86.miniroot-safe

#---------------------END BOOTADM--------------------

Also, don't forget, now you can boot from zfs files such as ...

bootsign

splash.xpm.gz

capability

menu.lst

Reside in /rpool/boot/grub, if you boot zfs and need to make a change to menu.lst either use bootadm (recommended method) or edit this one (not recommended).

This one is for booting from cdrom (it's a standard menu.lst entry when installing from cdrom/dvd).

title Solaris

        kernel /boot/multiboot kernel/unix -B install_media=cdrom

        module /boot/x86.miniroot

title Solaris Serial Console ttya

        kernel /boot/multiboot kernel/unix -B install_media=cdrom,console=ttya

        module /boot/x86.miniroot

title Solaris Serial Console ttyb (for lx50, v60x and v65x)

        kernel /boot/multiboot kernel/unix -B install_media=cdrom,console=ttyb

        module /boot/x86.miniroot

---o---

I was asked if I could add something about usb booting: this is what I have so far. It's good for getting a machine up of it's knees but it's not great for the usb device if it's a stick.

ZFS usb boot disk:

Go through your desired build of a zfs based system with at least network support, once done reboot the new build.

When the new build has booted, login and stop volume managent or it get's in the way of writing to the usb device use: svcadm disable -t system/filesystem/volfs

Find the usb device.

rmformat (format -e also shows the device but it might not be so obvious)

create a parttion

fdisk /dev/rdsk/<disk-from-rmformat>

slice the disk using format -e, give everything over to root, no need for lots of slices on a usb emergency boot disk

Once you have completed the above use "zpool attach rpool <original-disk> <usb-disk>". Then, once the resilvering has finished...

Install the master boot record.

fdisk -b /usr/lib/fs/ufs/mboot -n /dev/rdsk/c2t0d0p0

installgrub

installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c2t0d0s0

You might want to remove the splash screen stuff from /rpool/boot and menu.lst since it's a waste of space and load time. Then go for a trial boot. Since zfs doesn't use bootenv.rc to find it's boot device you should be able to boot this usb device easily. Be aware though that inserting this device could upset the device boot order; Also, since this is an o/s on a stick, you will be able to mount ufs as well as zfs disks. Don't forget the utilities as well, things like ssh, ftp, ipmitool etc are all useful and, if your planning on using this to send us information, they are essential.

---o---

Grub's really quite interesting for me so this will be another one I'll update regularly.

Sometimes questions come up on how to boot the machine, especially when the machine is having a bad day, This isn't a complete post by a long way yet but it will get you off to, what I believe, will be a good start. Try not to get over ambitious with grub to begin with, making mistakes on a lab/dev machine is better than a p45.

I suppose the prime assumption is that grub is already loaded, if not then read the grub manual (the link is in this text).

General Grub Stuff
---o---
once at the grub menu hit "c", this will drop you to the grub> command line interface from here you can:
1) find the most likely bootable disk

    find /boot/grub/stage1 or findroot (depending on your version of Solaris and grub)

This returns a list of possible boot disks/slices in the (<disk>,<partition>,<slice>) format

2) install grub itself (risky if your not familiar with grub)
    setup <disk> which is usually something like... hd0 hd1 etc
    or
    setup <disk,partition> which would look like hd0,0 for the first disk and the first partition on that disk
---o---

Booting whatever will boot
---o---
This is useful. I've taken this directly from the grub manual which can be found in the following link since I couldn't explain it better. The grub manual is definitely worth a good read.

http://www.gnu.org/software/grub/manual/grub.txt and is Copyright (C) 2004 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111, USA

Booting fallback systems
----------------------------

GRUB supports a fallback mechanism of booting one or more other entries if a default boot entry fails. You can specify multiple fallback entries if you wish.

Suppose that you have three systems, `A', `B' and `C'. `A' is a system which you want to boot by default. `B' is a backup system which is supposed to boot safely. `C' is another backup system which is used in case where `B' is broken.

Then you may want GRUB to boot the first system which is bootable among `A', `B' and `C'. A configuration file can be written in this way:

Remember though that for ufs and zfs the root (hd0,0) entry and all that it relies on becomes root (hd0,0,a) or rootfs or similar (disk,partition,slice), I've written some examples of this in the boot and the boot archive section.

default saved # This is important!!!
timeout 10
fallback 1 2 # This is important!!!

title A
root (hd0,0)
kernel /kernel
savedefault fallback # This is important!!!

title B
root (hd1,0)
kernel /kernel
savedefault fallback # This is important!!!

title C
root (hd2,0)
kernel /kernel
savedefault

Note that `default saved' (*note default::), `fallback 1 2' and `savedefault fallback' are used. GRUB will boot a saved entry by default and save a fallback entry as next boot entry with this configuration.

When GRUB tries to boot `A', GRUB saves `1' as next boot entry, because the command `fallback' specifies that `1' is the first fallback entry. The entry `1' is `B', so GRUB will try to boot `B' at next boot time.

Likewise, when GRUB tries to boot `B', GRUB saves `2' as next boot entry, because `fallback' specifies `2' as next fallback entry. This makes sure that GRUB will boot `C' after booting `B'.

It is noteworthy that GRUB uses fallback entries both when GRUB itself fails in booting an entry and when `A' or `B' fails in starting
up your system. So this solution ensures that your system is started even if GRUB cannot find your kernel or if your kernel panics.

However, you need to run `grub-set-default' (*note Invoking grub-set-default::) when `A' starts correctly or you fix `A' after it crashes, since GRUB always sets next boot entry to a fallback entry. You should run this command in a startup script such as `rc.local' to boot `A' by default:

# grub-set-default 0

where `0' is the number of the boot entry for the system `A'.

If you want to see what is current default entry, you can look at the file `/boot/grub/default' (or `/grub/default' in some systems). Because this file is plain-text, you can just `cat' this file. But it is strongly recommended *not to modify this file directly*, because GRUB may fail in saving a default entry in this file, if you change this file in an unintended manner. Therefore, you should use
`grub-set-default' when you need to change the default entry.
---o---

Grub and ZFS
---o---
To boot from a ZFS root filesystem, the kernel$ or module$ commands must include "-B $ZFS-BOOTFS" to expand to the zfs-bootfs boot path.

If you see zfs_open failures that may well mean that the pool is offline or a file in the pool is missing.

example menu.lst entries for zfs boot

title s10x_u6wos_07b_zfs
findroot (BE_s10x_u6wos_07b_zfs,0,a)
bootfs rpool/ROOT/s10x_u6wos_07b_zfs
kernel$ /platform/i86pc/multiboot -B $ZFS-BOOTFS
module /platform/i86pc/boot_archive

title s10x_u6wos_07b_zfs failsafe
findroot (BE_s10x_u6wos_07b_zfs,0,a)
bootfs rpool/ROOT/s10x_u6wos_07b_zfs
kernel /boot/multiboot kernel/unix -s
module /boot/x86.miniroot-safe

Zfs and ufs are supported in grub as provided by Sun, you cannot expect standard grub to do the same.
---o---

Grub Boot Stages
---o---

Grub Stage1:

This attempts to read bios and understand the disk geometry, it also finds the disk where stage2 can be loaded from including the sector is starts at. It's also here that you might norice the first disk error (i/o error of some sort). If you have made some sort of error in your bios regarding disk geometry (and not allowed the bios to read that information from the disk itself) you may also see sector read errors and a failure to load stage2. Always check bios if you see read beyond or sector based errors before logging a call.

Grub Stage2

Once stage1 has found the stage2 sector it hands over entirely to stage2. at this point grub attempts to setup cdrom boot emulation, if this is a hard disk then cdrom boot emulation is discarded. grub then goes on to assess the drive geometry again and also attempts to cope with a few peculiarities than the various bios manufacturers have. It's best not to believe that all bios's are equal and certainly do not assume you have a bug free bios (you almost certainly do not).

So, once grub stage2 has gotten past fixing the vagaries of the various bios it goes on to attempt it's own version of booting (that is, getting a image ready to run). this is where multiboot is also setup along with trying to understand the potential images to load and pass off to. grub stage2 make a really decent attempt at reading the images and it does take time to both read and decompress large image files. don't forget, these image files have a lot in them, they have the kernel itself, kernel modules and configuration files. almost always they are compressed (this is why the stage2 dot's take so much longer than stage1's).

The grub stage2 boot also sets up any video output, don't expect too many fancy things, there's a limited amount you can expect to find without operating system drivers available.

Here also grub stage2 makes the effort to work with both new and aged kernels and their issues, it also checks memory size too. don't forget though that the issues known about are as good as the last gub patch or installation you installed.

Grub also provides for editing of the command line used to load the image, this is important since it cannot possibly know all the ways in which you prefer to boot. this part can make or break a boot. it takes a lot to get here and if you do not understand or clumsily use some parameter that you have spied on the internet somewhere don't expect it to always work. the ethos here is that you "understand your kernel and the parameters you can use to boot in the most efficient way for you".
---o---

Almost certainly when a customer has a question about one of our x86 machines we need some information, Just as it is in the world of sparc it cannot be avoided. I thought I would post this little lot here as a starting point at least. These questions I ask nearly every time I have to provide the answer to the question "What's wrong with it".

Please be aware that we need a problem statement and we will dig for this, simply telling us that the machine does not work really does not help us to help you.

Most of the information we gather from the sp and for this we use a shell on the sp itself and also an opensource product called IPMITOOL. It's far too useful not to have installed somewhere on the same network as your problem machine (laptop, desktop etc) if it's allowed. 

Windows ipmitool: http://www.sun.com/download/products.xml?id=46f1ff04
Look here for the latest SUN  Supported ipmitool and search for ipmitool: http://www.sun.com/download/index.jsp?tab=2
Open source (We do not support using this one for data gathering)  ipmitool: http://ipmitool.sourceforge.net/

Always use the latest possible ipmitool but never anything less than 1.8.8.x (always update ipmitool to the very latest that you can manage).

Please bear in mind that you may need to update ipmitool or use one from the tools and drivers cd "As directed" by your support contact, This  will be because the newer the machine the more up to date you need your ipmitool version to be and vice versa. Please make sure you use the correct version, The better our information the better our diagnosis.

BIG NOTE:

You need to check that you have a support contract for Windows or the many Linux variants, It's not a given that you can expect support on these operating systems simply because they reside on Sun  Kit

Run the following against the Ilom:

ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> bmc info
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sel elist
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> fru
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sensor
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sunoem led get all

(on some of the newer machines (intel) you might get "Sun OEM Get LED command failed: Parameter out of range" if that happens use the following command  to get the led status (it just adds sb to led to form "sbled"):
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sunoem sbled get all

ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sdr elist
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sdr elist full
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> sdr elist all
ipmitool -I lan -H <ILOM IP address> -U <ILOM Username> chassis status

You might find that, if we are interested in something specific we might ask for more in depth ipmitool command output, please be patient when we do. A common additional switch we might ask for is for you to place a -v or a -vv or a -vvv directly after ipmitool (ipmitool -vvv)

On the Ilom itself
Ssh to the Ilom  as root and run the following:  (do this on the cmm as well if you have a blade chassis (and label it so we know it's the cmm and blade Ilom):

show  / -l  all

It does emit a lot of information, You would be advised to use the Solaris (or the Linux equivalent) "script" tool thus... at the prompt (when I say prompt, this means you will most likely be at another machine/terminal and not on the local problem machine) type : script -a <filename-of-your-choice> when done typing commands and gathering output type: ctrl-d and script will exit leaving you with <filename-of-your-choice> which you can email in.

And on the blade / host (if possible):

General none o/s tools
PcCheck is useful for testing the machine, It comes bundled on the tools and drivers cd which you can download from sun.com/download do remember to download the correct tools and drivers cd for your model type.

Memtest86 is a very good and useful tool if you think you might have memory problems too.
http://www.memtest.org/#downiso

Windows
if the machine is running Windows then the system and event log saved as text files, if it produced a memory dump then that as well. also  http://www.microsoft.com/downloads/details.aspx?familyid=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0&displaylang=en
and use those down loadable tools to send in various reports (you don;t have to install reporting tools for those facilities you don't have).

Redhat
if the machine is running Linux (RedHat) then the output of "sysreport" , did you also know that Redhat can produce crash dumps? if you use this link http://www.redhat.com/support/wpapers/redhat/netdump/  you can also send the dump to a networked machine (not your problem machine) which makes things much easier if we ask to see it. I might add instructions here later on on how to configure it (let me know if that would be useful to you). don't be put off by the documents age, it's still quite valid (make sure it is for your version of redhat obviously).

Suse

For the latest crash dump stuff under opensuse try http://en.opensuse.org/Kdump . If you want or need to use lkcd then look here for more information  http://www.novell.com/coolsolutions/feature/15284.html

Before sending us your suse crash dump for analysis make sure your running a supported version of suse for your machine type, if your not it's not not likely we can analyse the crash since we would need the "exact" same kernel environment you have to do the analysis, if your running some esoteric version of suse it's not likely we'll have it and you'll be out of luck.

install kdump (using yast2 kdump from the command line or install software from menu). kdump can be found on the distribution cd/dvd  enable settings as per your configuration (memory, filters etc) I can't tell you what the best settings are, you need to know that, make sure you don't use too much memory for the crash kernel though (trial and error).

Make sure that the dump target is realistic, i.e: it has enough space to consume the dump and can be mounted later in a emergency., you can set the target to be a filesystem ftp, nfs etc

Gather a supportconfig. if you can't find that then try http://en.opensuse.org/Supportutils  let me know if the link is either broken or useless for you. (opensuse you can install using the following command line: sudo zypper install supportconfig) so long as you have a yum server or internet connection it'll install for you).

Generic Linux
if the machine is running Linux then check for the presence of /var/log/mce and send those in. If it does not exist then please run the. command /usr/sbin/mcelog >> /var/log/mcelog and send in the resultant log file. if you do not have mcelog installed then you can get it here...
also collect the following... lspci -xxxx (gives us a pci dump we can interrogate using the -F (file) option)

http://rpmfind.net/linux/rpm2html/search.php?query=mcelog

If you using this on a amd opteron/athlon then the best command line is: mcelog --k8 --dmi --ascii >> /var/log/mcelog

Vmware
collect a vm-support script output (vm-support is installed as part of the default install) from the console of the server
use vmkdump -l <core-file-name> to extract the kernel log, also make sure you save the dump itself. either we or vmware can look through this for you.

gather the psod (use print screen, screen grab, digital camera if need be)
contact vmware themselves and request details on how to use the serial logging feature, basic instructions are update SerialPort to equal whichever port you have connected your laptop/other machine to (it's in advanced setting(s)), startup your favourite terminal software and start logging to / capturing the text to disk. when this is done or you have enough send them in.
once all this is gathered run the diagnostics which you can download for your specific machine/model from sun.com/download

Solaris
if the machine is running Solaris, Then  a explorer [and crash dump if one is produced].

Some questions to ask yourself first:
Sometimes there is no hardware problem, Sometimes a configuration change or patch or something else might be the real problem. It might pay to ask yourself the following questions before you log the call. I have a small list of questions I ask which, if nothing else, clarifies things a little. It also helps to think about priorities, You might have 2, 3 or more questions you want answers to; it pays then to prepare yourself by putting these questions in order of importance. Make the priority known to the person who takes your case on. Make sure that you have a clear "what is the problem with what" statement. Make sure that you clearly state your timescales and be clear about how this is affecting your project/business unit or whole company. Very often when logging a call it's possible to get caught up in a lot of other things, stay focussed and be precise, you would be surprised how much that helps.

1). did this ever work and if it did work, when was the last time it worked
2). do you have anything else the same as this that does work
3). what drew your attention to the problem (logging, monitoring). provide this information
4). is there more than one of the same thing that's broken
5). have you changed anything in the last four weeks (or more if the problem has been around longer than that and you have just decided to log the call) patching, configuration changes etc, be prepared to have your change logs handy 

Sometimes, after a component is replaced (and there are a lot of customer replaceable parts in the x86 kit) you will probably need to tell the operating system you have repaired and or replaced something (not all, but some (like Solaris) need you to). In Solaris you do this using fmadm faulty to find the problem component and fmadm repair to notify Solaris it need no longer concern itself over this item any more (use the man pages to find the full syntax for these commands). it's worth doing this since you may find the message logs or other notification paths telling you about faulty items again (the same one?) when it's not faulty.
---o---