We've been seeing an intermittent bug where when we have a mirrormount and we unmount the parent, it fails to do so. We don't know why, but we thought we had a reproducible test case. It was in the miniPIT test suite. The problem with that is that it is huge and thus hard to debug.

At a first blush, this is easy to conceptualize:

[root@pnfs-18-09 ~]> mount -o vers=4 pnfs-18-11:/a_mnt /mnt
[root@pnfs-18-09 ~]> cd /mnt
[root@pnfs-18-09 /mnt]> cd stress/
[root@pnfs-18-09 stress]> nfsstat -m `pwd`
/mnt/stress from pnfs-18-11:/a_mnt/stress
 Flags:         vers=4,proto=tcp,sec=sys,hard,intr,link,symlink,acl,mirrormount,rsize=1048576,wsize=1048576,retrans=5,timeo=600
 Attr cache:    acregmin=3,acregmax=60,acdirmin=30,acdirmax=60

[root@pnfs-18-09 stress]> cd 
[root@pnfs-18-09 ~]> umount /mnt

We see that the mirrormount did succeed, but we also see that the unmount did as well.

It turned out that the test case we had was on UFS and not ZFS. I was wondering how the test suite could easily create a myriad of UFS filesystems on the fly - after all, there aren't that many slices available. My colleague Helen Chao showed me how we do it with mkfile and lofiadm:

[root@pnfs-18-11 ~]> mkfile 1G /export/FILE2
[root@pnfs-18-11 ~]> mkfile 1G /export/FILE1
[root@pnfs-18-11 ~]> lofiadm -a /export/FILE1
/dev/lofi/1
[root@pnfs-18-11 ~]> lofiadm -a /export/FILE2
/dev/lofi/2
[root@pnfs-18-11 ~]> newfs /dev/lofi/1
newfs: construct a new file system /dev/rlofi/1: (y/n)? y
/dev/rlofi/1:   2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors
        1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632,
 2083232, 2092832
[root@pnfs-18-11 ~]> newfs /dev/lofi/2
newfs: construct a new file system /dev/rlofi/2: (y/n)? y
/dev/rlofi/2:   2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors
        1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632,
 2083232, 2092832
[root@pnfs-18-11 ~]> mkdir /a_mnt
[root@pnfs-18-11 ~]> mount /dev/lofi/1 /a_mnt
[root@pnfs-18-11 ~]> mkdir /a_mnt/stress
[root@pnfs-18-11 ~]> mount /dev/lofi/2 /a_mnt/stress
[root@pnfs-18-11 ~]> share -F nfs /a_mnt
[root@pnfs-18-11 ~]> share -F nfs /a_mnt/stress

I probably overkilled the file size in the mkfile since I'm not writing to them, but in all, this is a very neat piece of hackery she does here. I really like how she follows the Unix paradigm of using small tools to do complex tasks.