Can't use a zvol as the boot vdsdev with Solaris 10 5/08
Well, it seems like my dreams of using a vdsdev with a zvol backend as a boot-device for Solaris 10 5/08 have been dashed.
Here's my environment:
- T5220
- Solaris 10 5/08 127127-11
- LDoms 1.0.3
- Firmware 7.1.6.d
- Guest domain
- Installed via JET
- Flar-based install from same Solaris 10 5/08 image as the T5220
While it is possible to present a zvol to a guest domain as a single slice, Solaris 10 5/08 does not support a single-slice install. So you could install on a whole disk, or on an image file from within a zfs dataset, and then present more storage after the jump. But you can't use a zvol as your boot-device.
This problem seems to be fixed with Solaris 10 10/08:
Before Solaris 10 10/08 (Update 6), the installation of Solaris is possible only on a full disk, not on a single-slice disk.
Starting with Solaris 10 10/08 (Update 6), the installation of Solaris is also possible on a single-slice disk. In that case, if the Solaris installation is done using the UFS filesystem then only the root partition must be defined and this partition must use all the disk space.
Next week I'll be working on our 10/08 build, so I'll be able to give this a try. 10/08 supports a zfs root, which is essential -- more on that in a moment.
So why do I care about a zvol boot-device? LUNs. Say you have a big LUN presented to your control domain. ZFS is an easy and effective way of carving up the LUN and giving storage to your guest / service / I/O domains. You get block-level snapshots, easy administration, and pretty decent performance. Oh, and it's free too. But without zvols you have to make an image file to use as a backend for your vdsdev, i.e.,
# zfs create lunpool/guestldom
# mkfile 36g /lunpool/guestldom/imgfile
# ldm add-vdsdev /lunpool/guestldom/imgfile guestldom-system-disk@primary-vds0
This is all well and good. But mkfile pads the image file that you create with 0s. This can take quite a while. Sure, you could do it once, and then snapshot and clone the containing dataset. But then all of your image files would be dependent upon that initial snapshot. Not a huge deal, but it could be annoying.
Zvols do a bit to address this issue, but not completely. You can create a zvol of arbitrary size very quickly, i.e.,
# zfs create -V 36g lunpool/guestldom
There are no 0s to be written, the command comes back in a snap. But according to the LDoms cookbook, your guest domain will need to be installed with a single filesytem mounted at /. This is a big problem if we're talking UFS. For example, how will you isolate, say, /var from /kernel? You could maybe use some fs quotas, but that's not really ideal. And what about swap? I have no idea how that will work -- with only s0 will swap live in a file on top of UFS? That'll be interesting to see; I'll post about that next week when I get to try it.
A zfs root could solve this problem. /var can be isolated from /. Swap gets its own dataset. I'm a little worried about where the zfs database will live, I'll have to see.
So... I've still got more questions than answers. But one thing is clear: upgrade to 10/08!
Why would you ever want to present a ZVOL as a single-slice vdsdev? Just present it as a regular vdsdev and be done with it.
Single-slice vdsdevs should really only be used if you're utilizing physical back-end disk slices with existing data on them that you want to give to a LDOM without having to mess with the data.
If you create a ZVOL, and present it as a regular vdsdev (ie. not-single slice) you can use that to install to. I just did this with a Solaris 10 Update 5 (05/08) with no additional patches for both the Host and Guest without any issues.
Posted by Phillip Wagstrom on December 18, 2008 at 04:32 PM EST #
Oh, believe me, I'd much rather present the zvol as a whole disk. Whenever I try to do so I get the 'no disks' error on jumpstart. Also, a prtvtoc on the zvol returns with only s0.
I had read that 5/08 can only export zvols as single-slice, and with the installation troubles I assumed that to be true.
Man, I'd love to get this working with 5/08. Which kernel are you running? I believe mine is 127127-11.
Posted by Jason Callaway on December 18, 2008 at 07:02 PM EST #
A zvol should work with S10U5/127127-11. Have you looked at the LDoms Admin Guide ?
The LDoms 1.0.3 version is at http://docs.sun.com/source/820-4894-10/chapter5.html
e.g. this should do the trick
# ldm add-vdsdev /dev/zvol/dsk/lunpool/guestldom guestldom-system-disk@primary-vds0
Posted by Liam Merwick on December 19, 2008 at 10:08 AM EST #
I've got this in a lab system on SWAN if you want to poke around, but I just did this from scratch using Solaris 10 05/08 without any additional patches. 127127-11 comes with that OS.
The only think that I can think of is that you're using the raw interface instead of the block interface. You should be using the /dev/zvol/dsk/poolname/volname as the device.
Works without a problem for me. I'd be happy to look at your config if its on SWAN as well.
Posted by Phillip Wagstrom on December 19, 2008 at 10:18 AM EST #
Thanks for the comments, guys, I really appreciate the help.
@Liam, yeah I've looked at the admin guide. When I create the vdsdev I do it just as you describe. However when I try to jump the guest domain, I get that 'no disks cabled or powered up' error. Also when I prtvtoc the zvol I see only s0. So that's why I was assuming that the zvol was being exported as single-slice.
@Phillip, thanks for the offer to take a look. Unfortunately I'm on a private LAN, and not SWAN. My vdsdev is using the naming convention you mention, and I'm not using the raw device.
The good news is that I'm in charge of the baseline, so if I can't fix this, I can always cut a new baseline with 10/08...
Posted by Jason Callaway on December 19, 2008 at 02:06 PM EST #