Jerry Jelinek's Weblog: Jerry Jelinek's Weblog

I am one of the engineers working on Solaris Volume manager. There are some questions I get asked a lot and I thought it would be useful to get this information out to a broader audience so I am starting this blog. We have also done a lot of work on the volume manager in the recent Solaris releases and I'd like to talk about some of that too.

One of the questions I get asked most frequently is how to do root mirroring on x86 systems. Unfortunately our docs don't explain this very well yet. This is a short description I wrote about root mirroring on x86 which hopefully explains how to set this up.

Root mirroring on x86 is more complex than is root mirroring on SPARC. Specifically, there are issues with being able to boot from the secondary side of the mirror when the primary side fails. Compared to SPARC systems, with x86 machines you have be sure to properly configure the system BIOS and set up your fdisk partitioning.

The x86 BIOS is analogous to the PROM interpreter on SPARC. The BIOS is responsible for finding the right device to boot from, then loading and executing the master boot record from that device.

All modern x86 BIOSes are configurable to some degree but the discussion of how to configure them is beyond the scope of this post. In general you can usually select the order of devices that you want the BIOS to probe (e.g. floppy, IDE disk, SCSI disk, network) but you may be limited in configuring at a more granular level. For example, it may not be possible to configure the BIOS to boot from the first and second IDE disks. These limitations may be a factor with some hardware configurations (e.g. a system with two IDE disks that are root mirrored). You will need to understand the capabilities of the BIOS that is on your hardware. If your primary boot disk fails and your BIOS is set up properly, then it will automatically boot from the second disk in the mirror. Otherwise, you may need to break into the BIOS while the machine is booting and reconfigure to boot from the second disk or you may even need to boot from a floppy with the Solaris Device Configuration Assistant (DCA) on it so that you can select the alternate disk to boot from.

On x86 machines fdisk partitions are used and it is common to have multiple operating systems installed. Also, there are different flavors of master boot programs (e.g. LILO or Grub), in addition to the standard Solaris master boot program. The boot(1M) man page is a good resource for a detailed discussion of the multiple components that are used during booting on Solaris x86.

Since SVM can only mirror Solaris slices within the Solaris fdisk partition this post focuses on a configuration that only has Solaris installed. If you have multiple fdisk partitions then you will need to use some other approach to protect the data outside of the Solaris fdisk partition. SVM can't mirror that data.

For an x86 system with Solaris installed there are two common fdisk partitioning schemes. One approach uses two fdisk partitions. There is a Solaris fdisk partition and another, small fdisk partition of about 10MB called the x86 boot partition. This partition has an Id value of 190. The Solaris system installation software will create a configuration with these two fdisk partitions as the default. The x86 boot partition is needed in some cases, such as when you want to use live-upgrade on a single disk configuration, but it is problematic when using root mirroring. The Solaris system installation software only allows one x86 boot partition for the entire system and it places important data on that fdisk partition. That partition is mounted in the vfstab with this entry:

/dev/dsk/c2t1d0p0:boot - /boot pcfs - no -

Because this fdisk partition is outside of the Solaris fdisk partition it cannot be mirrored by SVM. Furthermore, because there is only a single copy of this fdisk partition it represents a single point of failure.

Since the x86 boot partition is not required in most cases it is recommended that you do not use this as the default when setting up a system that will have root mirroring. Instead, just use a single Solaris fdisk partition and omit the x86 boot partition for your installation. It is easiest to do this at the time you install Solaris. If you already have Solaris installed and you created the x86 boot partition as part of that process then the easiest thing would be to delete that with the fdisk(1M) command and reinstall, taking care not to create the x86 boot partition during the installation process.

Once your system is installed you create your metadbs and root mirror using the normal procedures.

When you use fdisk to partition your second disk you must make the disk bootable with a master boot program. The -b option of fdisk(1M) does this.

e.g. fdisk -b /usr/lib/fs/ufs/mboot /dev/rdsk/c2t1d0p0

On x86 machines the Solaris VTOC (the slices within the Solaris fdisk partition) is slightly different from what is seen on SPARC. On SPARC there are 8 VTOC slices (0-7) but on x86 there are more. In particular slice 8 is used as a "boot" slice. You will see that this slice is 1 cylinder in size and starts at the beginning of the disk (offset 0). The other slices will come after that, starting at cylinder 1.

Slice 8 is necessary for booting Solaris from this fdisk partition. It holds the partition boot record (pboot), the Solaris VTOC for the disk and the bootblk. This information is disk specific so it is not mirrored with SVM. However, you must ensure that both disks are bootable so that you can boot from the secondary disk if the primary fails. You can use the installboot program to setup the second disk as a Solaris bootable disk (see installboot(1M)). An example command is:

installboot /usr/platform/i86pc/lib/fs/ufs/pboot \ /usr/platform/i86pc/lib/fs/ufs/bootblk /dev/rdsk/c2t1d0s2

There is one further consideration for booting an x86 disk. You must ensure that the root slice has a slice tag of "root" and the root slice must be slice 0. See format(1M) for checking and setting the slice tag field.

Solaris x86 emulates some of the behavior of the SPARC eeprom. See eeprom(1M). The boot device is stored in the "bootpath" property that you can see with the eeprom command. The value is the device tree path of the primary boot disk. For example:

bootpath=/pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@0,0:a

You should set up the alternate bootpath via the eeprom command so that the system will try to boot from the second side of the root mirror. First you must get the device tree path for the other disk in the mirror. You can use the ls command. For example:

# ls -l /dev/dsk/c2t1d0s0 lrwxrwxrwx 1 root root 78 Sep 28 23:41 /dev/dsk/c0t1d0s0 -> ../../devices/pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@1,0:a

The device tree path is the portion of the output following "../devices".

Use the eeprom command to set up the alternate boot path. For example:

eeprom altbootpath='/pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@1,0:a'

If your primary disk fails you will boot from the secondary disk. This may either be automatic if your BIOS is configured properly, you may need to manually enter the BIOS and boot from the secondary disk or you may even need to boot from a floppy with the DCA on it. Once the system starts to boot it will try to boot from the "bootpath" device. Assuming the primary boot disk is the dead disk in the root mirror the system will attempt to boot from the "altbootpath" device.

If the system fails to boot from the altbootpath device for some reason then you need to finish booting manually. The boot should drop in to the Device Configuration Assistant (DCA). You must choose the secondary disk as the boot disk within the DCA. After the system has booted you should update the "bootpath" value with the device path that you used for the for the secondary disk (the "altbootpath") so that the machine will boot automatically.

For example, run the following to set the boot device to the second scsi disk (target 1):

eeprom bootpath='/pci@0,0/pci8086,2545@3/pci8086,1460@1d/pci8086,341a@7,1/sd@1,0:a'

Note that all of the usual considerations of mddb quorum apply to x86, just as they do for SPARC.