Thursday June 19, 2008

JET and SVM, a match made in heaven...

The flar that we created earlier now has all the packages and patches (oh yeah, JET patches the system for you as well) in it that we want, and has the basic filesystems installed for us on the root disk. We implemented the root disk layout (reserving s7 for metadb in the SVM sections later) with this section of the JET template:

# Define Root disk configuration # Make sure that /var has a Live Upgrade slice # if /var is a separate filesystem. # # Mount the LU spaces to make sure that someone # doesn't come along later and use that "free space". # # s6 is the "freehog" partition to contain all free # space after the static slices are allocated # # s7 is defined later and used as a metadb space. # base_config_profile_usedisk=rootdisk. base_config_profile_dontuse="" base_config_profile_root="8192" base_config_profile_swap="32768" base_config_profile_s3_mtpt="/GZ_VAR_LU" base_config_profile_s3_size="8192" base_config_profile_s4_mtpt="/GZ_ROOT_LU" base_config_profile_s4_size="8192" base_config_profile_s5_mtpt="/var" base_config_profile_s5_size="8192" base_config_profile_s6_mtpt="" base_config_profile_s6_size="free" base_config_profile_s7_mtpt="" base_config_profile_s7_size=""

Hint number one... Put lots of comments in your template files to remind yourself (and others who come along later) why and how you did things. These template files can be rather large and complex.

This section defines the rootdisk (reserved word in JET) with a / partition of 8G on s0, swap space of 32G on s1, /var of 8G on s5, and some Live Upgrade partitions for / and /var on s3 and s4 with matching sizes (important). The metadb space for SVM will be on s7 (defined later in the template), and any leftover space will be allocated to a partition on s6, but not mounted. We will add this free space to our pile of space for use by zones later on.

# # Any devices we need to skip (spare slices on root disks etc) # # Format: c?t?d?s? # sds_skip_devices="c1t0d0s6 c1t1d0s6"

This tells JET that we don't want to use the root mirroring phase to set up the partitions that we will use later as part of the zone disk space.

# # Additional disks under JET control. Skip the # root mirror disk (c1t1) as it is defined in the # mirroring steps below. # base_config_profile_additional_disks="c1t2d0 c1t3d0 \ c1t4d0 c1t5d0 c1t6d0 c1t7d0 "

This is where comments come into play. Without this comment, you might think that the root mirror disk should be listed in "additional disks under JET control", it just makes sense. But no, that would make very ugly things happen.

Now that we have reserved the disks for JET to use, we need to layout a disk partition scheme for them. Again, we don't want to mount them, and we will use s7 later on to add metadbs for SVM to use:

# # Define layout of the additional disks to use # s6 as the freehog space. s7 will have already # been reserved by the metadb definitions below. # base_config_profile_disk_c1t2d0s6_mtpt="" base_config_profile_disk_c1t2d0s6_size="free" base_config_profile_disk_c1t3d0s6_mtpt="" base_config_profile_disk_c1t3d0s6_size="free" base_config_profile_disk_c1t4d0s6_mtpt="" base_config_profile_disk_c1t4d0s6_size="free" base_config_profile_disk_c1t5d0s6_mtpt="" base_config_profile_disk_c1t5d0s6_size="free" base_config_profile_disk_c1t6d0s6_mtpt="" base_config_profile_disk_c1t6d0s6_size="free" base_config_profile_disk_c1t7d0s6_mtpt="" base_config_profile_disk_c1t7d0s6_size="free"

That sets up all of our physical disks (except for the metadb stuff, but that will come along later). At this point, we could install the machine and make sure that all is well, and our rootdisk works as expected. The next step is to add in the SVM stuff. Let me repeat, now is a REALLY good time to stop, try things out, and make sure that your root disk and "additional disks" are configured the way you want them.

There are three basic pieces in the SVM configuration that we need to worry about. We need to set up the metadb copies, copy the configuration of the rootdisk and mirror it, and then we need to set up the leftover diskspace and put it into a big metadevice to use later for zone space soft partitions.

# Kernel options # # If you need to increase the number of metasets from the # default (4) or the number of metadevices per metaset from # the default (128), then enter the figures here. # # Please note that increasing these numbers can significantly # increase the time taken to do a reconfiguration boot, or a # drvconfig etc. # sds_nummetasets="4" sds_nummetadevices="4000"

We needed to increase the default number of metadevices from 128 to something higher. We did some testing, and making this number in the thousands didn't hurt our performance, so we erred on the side of safety with 4000.

# This variable defines where SVM will create the metadbs. # If any meta state databases are to be created, add # the c?t?d?s? number here. If you need multiple copies # (i.e. metadb -c 3), suffix the device with a : and the # number of copies. e.g. c0t0d0s7:3 # # eg: sds_database_locations="c3t0d0s7:3 c1t0d0s7:3" # sds_database_locations="rootdisk.s7:3 c1t2d0s7:3 c1t3d0s7:3 \ c1t4d0s7:3 c1t5d0s7:3 c1t6d0s7:3 c1t7d0s7:3"

In theory, according to the template comments, you don't need to specify the rootdisk or the root mirror disk in this variable. We didn't notice the comment until we already had a working configuration, and left things alone. Your mileage may vary, but we don't get warnings or errors with this configuration and everything is working fine for us.

We have set up metadb partitions in this step, and placed three copies on each metadb partition. Again, this was a part of the build specification, and traditional for the customer. That is 24 copies of the metadb, and I am not sure if I would have configured things this way if the choice was mine. Definitely do your due diligence and make your own configuration decisions wisely. Your mileage may vary.

# # This variable ensures that partitions are created to # hold the metadbs defined above. # # Specify locations in one of the following forms # s:size - creates s7 on the rootdisk # c?t?d?s:size - creates slice on specified device # sds_database_partition="s7:32 c1t2d0s7:32 c1t3d0s7:32 \ c1t4d0s7:32 c1t5d0s7:32 c1t6d0s7:32 c1t7d0s7:32

We are creating eight metadb partitions, one on each disk, 32MB each. This section also reserves the proper space on s7 of the rootdisk. We had issues with the configuration when we tried to configure the metadb partition on s7 in the section of the template where we defined the rest of the rootdisk partitions. JET is smart enough to slice s7 out for us before calculating the "free" space for s6 on the root disk.

# If the boot device alias needs setting then do it here # ie sds_root_alias="rootdisk" # This will update the boot-device filed to ${sds_root_alias} # net and add the name to devalias, removing any previous # one of the same name sds_root_alias="rootdisk" ############ # # If we do have a root mirror, then set the devalias device # to this name # sds_root_mirror_devalias_name="rootmirr" ############ # # If you are using a two disk system and are mirroring the # root device, # you may want to enable md:mirrored_root_flag in the # kernel (/etc/system). # # You should read the info doc about this and fully understand # the implications of setting it... i.e. it's not just a case # of always setting it! # sds_mirrored_root_flag="1"

Here we assign a name for the root mirror disk. This will be the alias used in the boot prom to set up the root mirror as a second bootable device. We also set the md:mirrored_root_flag in /etc/system. Definitely read the Infodoc that the template mentions and make your own decision on this one.

# # By default, the root disk will be mirrored slice by # slice; the metadevices will start with d10 for the # first slice (sub-mirrors d11 and d12), d20 for # the second slice (sub-mirrors d21 and d22) upwards. # # If you wish to use your own numbering scheme for the # metadevices, please specify them here, in the following # format- # :mirror md:sub mirror 1:sub mirror 2 # sds_device_numbers="/:d0:d91:d92 swap:d1:d11:d12 /GZ_VAR_LU:d3:d31:d32 /GZ_ROOT_LU:d4:d41:d42 /var:d5:d51:d52"

In this section, we are defining the device names for the partitions of the root disk. We were following a local numbering scheme, and this worked well for us. One interesting note here that took a couple hours to debug, apparently metadevices may be named "d0", but they can't be named d0[anything]. So d01 is not allowed. d001 is not allowed. d02843 is not allowed. Oops. I'll definitely remember that one.

# If the root device is to be mirrored, define the mirror # here (c?t?d?). # # sds_use_fmthard can be set to "yes" | "no" | "" # # If sds_use_fmthard is set to "yes", then JET will create # metadb partitions and create the metadb as defined on # the root disk. You DO NOT need to specify the root mirror # in the sds_database_partition nor sds_database_locations # variable. # # If sds_use_fmthard is set to "no" or "", then JET will # create the data partitions for you, but you will have # to populate the sds_database_* variables if you want a # metadb to exist on the root mirror. # # You MUST set fmthard=yes for Solaris 9 and above. sds_root_mirror="c1t1d0" sds_use_fmthard=yes

Wow. That section is easy. Those two lines are all it takes to mirror the root disk. Just tell JET what disk to mirror it to, and tell JET to use "fmthard" to copy the partition table. This causes the installation to run a "prtvtoc" command against the configured root disk, and then feed that output to "fmthard -s" against the disk defined in sds_root_mirror. Simple.

For our configuration though, we will add specific naming for the root disk and root mirror disk partitions and metadevice names:

# sds_metadevices_to_init="d81:1:1:/dev/dsk/c1t0d0s0 d80:-m:d81" # # Equivalent to # metainit d81 1 1 /dev/dsk/c1t0d0s0 # metainit d80 -m d81 # # This will create a one-way mirror on d80 to d81. # # Example of combined "md.tab" and "command line" syntax: # (this would all be on one line, but has been split for # clarity) # # sds_metadevices_to_init="d71 d72 d70 d81:1:1:/dev/dsk/c1t0d0s0 # d82:1:1:/dev/dsk/c2t0d0s0 # d80:-m:d81 d91 d92 d90" sds_metadevices_to_init="d91 d92 d0 d91:1:1:/dev/dsk/c1t0d0s0 d92:1:1:/dev/dsk/c1t1d0s0 d0:-m:d91 d11 d12 d1 d11:1:1:/dev/dsk/c1t0d0s1 d12:1:1:/dev/dsk/c1t1d0s1 d1:-m:d11 d31 d32 d3 d31:1:1:/dev/dsk/c1t0d0s3 d32:1:1:/dev/dsk/c1t1d0s3 d3:-m:d31 d41 d42 d4 d41:1:1:/dev/dsk/c1t0d0s4 d42:1:1:/dev/dsk/c1t1d0s4 d4:-m:d41 d51 d52 d5 d51:1:1:/dev/dsk/c1t0d0s5 d52:1:1:/dev/dsk/c1t1d0s5 d5:-m:d51"

There we have it. Those pieces of the template define the metadbs and the root disk mirroring, and setup the partitions that we will use later to create the space for our zones. Fairly painless and straightforward, and definitely easier and safer than doing all of the twiddly bits by hand. Absolutely easier than repeating those manual tasks for 100 servers too! Next entry, I'll delve deeper into the zone partition space allocation, and the soft partitioning that we used to accomplish that piece.

This, again, is a REALLY good time to stop and try the template out. At this point, we have sliced up the root disk, created metadbs, sliced up the extra disks, and mirrored the root disk. These are the really tricky parts that will break a machine in interesting ways that are more difficult to debug. If we know that these parts are working, then creating the spaces on our leftover partitions for the zones will happen on a running and (hopefully) stable system environment, making debugging much easier.

bill.