Neat blogs
Navigation
Editing
statcounter.com
clustrmaps.com
technorati.com
Monday March 27, 2006 In doing [Open]Solaris' pseudo-fs, I got very frustrated in trying to share an export to a single host by IP. While I used syntax which I learned for Data ONTAP and Linux, my take-away is that the tools available did little to help me zero in on the issues at hand.
Reading the companion piece will make some of the dialog flow better.
The original set of shares was:
[tdh@wont isos]> share - /export/zfs rw "" - /export/zfs/nfsv2 rw "" - /export/zfs/nfsv3 rw "" - /export/zfs/tdh rw "" - /zoo/isos/mnt/freebsd/d2 rw "" - /zoo/isos/mnt/Fedora/dvd ro "" - /export/zfs/nfsv4 rw "" - /export/zfs/spud rw "" - /export/zfs/haynest rw "" - /export/zfs/coach rw "" - /export/zfs/loghyr rw "" - /export/zfs/mrx rw "" - /export/zfs/thomas rw ""
And I wanted to add a simple hare to just one machine as /zoo/isos:
[tdh@wont isos]> sudo share -F nfs -o rw=192.168.3.0/24 -d "For kanigix only" /zoo/isos
And try to get back in there:
[tdh@kanigix ~]> sudo umount -f /net/wont/zoo/isos [tdh@kanigix ~]> cd /net/wont [tdh@kanigix wont]> ls -la total 4 dr-xr-xr-x 3 root root 3 Mar 27 15:14 . dr-xr-xr-x 2 root root 2 Mar 27 15:14 .. dr-xr-xr-x 1 root root 1 Mar 27 15:14 export dr-xr-xr-x 2 root root 2 Mar 27 15:14 zoo [tdh@kanigix wont]> cd zoo [tdh@kanigix zoo]> ls -la total 3 dr-xr-xr-x 2 root root 2 Mar 27 15:14 . dr-xr-xr-x 3 root root 3 Mar 27 15:14 .. dr-xr-xr-x 1 root root 1 Mar 27 15:14 isos [tdh@kanigix zoo]> cd isos/ isos/: I/O error.
Okay, the I/O error is the 10% drop for NFS traffic on my firewall. But:
[tdh@kanigix zoo]> sudo mount wont:/zoo/isos /mnt [tdh@kanigix zoo]> ls -la /mnt total 7 drwxr-xr-x 10 tdh sys 10 Mar 27 14:52 . drwxr-xr-x 26 root root 512 Mar 27 13:27 .. drwxr-xr-x 4 tdh staff 4 Mar 22 21:30 mnt
Lets relax the share even further:
[tdh@wont isos]> sudo share -F nfs -o rw -d "For kanigix only" /zoo/isos
And try again:
[tdh@kanigix ~]> sudo umount /mnt [tdh@kanigix ~]> sudo mount wont:/zoo/isos /mnt [tdh@kanigix ~]> ls -la /mnt total 21 drwxr-xr-x 10 tdh sys 10 Mar 27 14:52 . drwxr-xr-x 26 root root 512 Mar 27 13:27 .. dr-xr-xr-x 3 root root 3 Mar 27 15:22 .zfs drwxr-xr-x 3 tdh staff 3 Mar 22 17:01 Fedora drwxr-xr-x 2 tdh staff 4 Mar 21 21:29 FreeBSD drwxr-xr-x 3 tdh staff 3 Mar 22 11:35 bfus drwxr-xr-x 2 root root 3 Mar 21 16:16 dontcare drwxr-xr-x 2 tdh staff 2 Mar 27 14:52 foo drwxr-xr-x 4 tdh staff 4 Mar 22 21:30 mnt drwxr-xr-x 2 root root 4 Mar 21 16:28 sparc drwxr-xr-x 2 root root 6 Mar 21 20:57 x86
What is going on here? Lets remove the pseudofs by going strictly to NFSv3:
[tdh@kanigix ~]> sudo mount -o vers=3 wont:/zoo/isos /mnt nfs mount: wont:/zoo/isos: Permission denied [tdh@kanigix ~]> snoop -i k2m3.snoop 1 0.00000 kanigix -> wont.internal.excfb.com PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 proto=UDP 2 0.00054 wont.internal.excfb.com -> kanigix PORTMAP R GETPORT port=51896 3 0.00029 kanigix -> wont.internal.excfb.com MOUNT3 C Null 4 0.00029 wont.internal.excfb.com -> kanigix MOUNT3 R Null 5 0.00101 kanigix -> wont.internal.excfb.com MOUNT3 C Mount /zoo/isos 6 0.00045 wont.internal.excfb.com -> kanigix MOUNT3 R Mount Permission denied
So it is from the mount protocol. We need a snoop trace from the server:
[tdh@wont ~]> snoop -i k2w3.snoop 1 0.00000 kanigix.internal.excfb.com -> wont PORTMAP C GETPORT prog=100005 (MOUNT) vers=3 proto=UDP 2 0.00032 wont -> kanigix.internal.excfb.com PORTMAP R GETPORT port=51896 3 0.00050 kanigix.internal.excfb.com -> wont MOUNT3 C Null 4 0.00011 wont -> kanigix.internal.excfb.com MOUNT3 R Null 5 0.00123 kanigix.internal.excfb.com -> wont MOUNT3 C Mount /zoo/isos 6 0.00080 wont -> kanigix.internal.excfb.com MOUNT3 R Mount Permission denied [tdh@wont ~]> host kanigix.internal.excfb.com kanigix.internal.excfb.com has address 192.168.3.109 [tdh@wont ~]> share | grep isos - /zoo/isos/mnt/freebsd/d2 rw "" - /zoo/isos/mnt/Fedora/dvd ro "" - /zoo/isos rw=192.168.3.0/24 "For kanigix only"
Lets try the following share:
[tdh@wont isos]> sudo share -F nfs -o rw=kanigix.internal.excfb.com -d "For kanigix only" /zoo/isos
And we get:
[tdh@kanigix ~]> sudo mount -o vers=3 wont:/zoo/isos /mnt [tdh@kanigix ~]> ls -la /mnt total 21 drwxr-xr-x 10 tdh sys 10 Mar 27 14:52 . drwxr-xr-x 26 root root 512 Mar 27 13:27 .. dr-xr-xr-x 3 root root 3 Mar 27 15:31 .zfs drwxr-xr-x 3 tdh staff 3 Mar 22 17:01 Fedora drwxr-xr-x 2 tdh staff 4 Mar 21 21:29 FreeBSD drwxr-xr-x 3 tdh staff 3 Mar 22 11:35 bfus drwxr-xr-x 2 root root 3 Mar 21 16:16 dontcare drwxr-xr-x 2 tdh staff 2 Mar 27 14:52 foo drwxr-xr-x 4 tdh staff 4 Mar 22 21:30 mnt drwxr-xr-x 2 root root 4 Mar 21 16:28 sparc drwxr-xr-x 2 root root 6 Mar 21 20:57 x86
Okay, something somewhere is just plain busted. Alright, reading the man pages, they state for an access list:
network The network or subnet component is
preceded by an at-sign (@). It can
be either a name or a dotted
address. If a name, it is converted
to a dotted address by
getnetbyname(3SOCKET). For example,
=@mynet
would be equivalent to:
=@172.16 or =@172.16.0.0
Does the '@' sign work for the subnet?
[tdh@wont isos]> sudo share -F nfs -o rw=@192.168.3.0/24 -d "For kanigix only" /zoo/isos
Yes. How about to just the one IP:
[tdh@wont isos]> sudo share -F nfs -o rw=@192.168.3.109 -d "For kanigix only" /zoo/isos
No:
[tdh@kanigix ~]> sudo mount -o vers=3 wont:/zoo/isos /mnt nfs mount: wont:/zoo/isos: Permission denied
Can we get tricky?
[tdh@wont isos]> sudo share -F nfs -o rw=@192.168.3.109/32 -d "For kanigix only" /zoo/isos
Which works.
And we can go back and show that it also works for NFSv4.
Summary
All of the tools, showmount and share, gave back to me output which indicated that everything was hunky-dory:
[tdh@wont isos]> sudo share -F nfs -o rw=192.168.3.0/24 -d "For kanigix only" /zoo/isos [tdh@wont isos]> share | grep kanigix - /zoo/isos rw=192.168.3.0/24 "For kanigix only" [tdh@wont isos]> sudo share -F nfs -o rw=192.168.3.109 -d "For kanigix only" /zoo/isos [tdh@wont isos]> share | grep kanigix - /zoo/isos rw=192.168.3.109 "For kanigix only" [tdh@wont isos]> sudo share -F nfs -o rw=@192.168.3.109 -d "For kanigix only" /zoo/isos [tdh@wont isos]> share | grep kanigix - /zoo/isos rw=@192.168.3.109 "For kanigix only" [tdh@wont isos]> sudo share -F nfs -o rw=@192.168.3.109/32 -d "For kanigix only" /zoo/isos [tdh@wont isos]> share | grep kanigix - /zoo/isos rw=@192.168.3.109/32 "For kanigix only"
And on the client:
showmount -e wont | grep "/zoo/isos" /zoo/isos 192.168.3.0/24 [tdh@kanigix isos]> showmount -e wont | grep "/zoo/isos" /zoo/isos 192.168.3.109 [tdh@kanigix isos]> showmount -e wont | grep "/zoo/isos" /zoo/isos @192.168.3.109 [tdh@kanigix isos]> showmount -e wont | grep "/zoo/isos" /zoo/isos @192.168.3.109/32
The '@' symbol is not in any published standard, Data ONTAP uses it to denote netgroups. But even the first 2 outputs read to me like the machine '192.168.3.109' is allowed access. In reality, it is the host with the name of 192.168.3.109 or the netgroup with the name of 192.168.3.109 which is allowed access. Also, having to add '/32' to mean a single machine is not intuitive.
Technorati Tags: NFSv4 OpenSolaris share_nfs
Earlier (OpenSolaris does not mount across filesystems, especially when they are not exported!), I wrote about how [Open]Solaris does not automatically mount via NFSv4. I want to examine that statement with some more examples.
So the exports on that server, kanigix, are:
[tdh@kanigix wont]> share - / rw "root" - /export/home rw "home"
And if I go in via explicit mounts, /export/home is not mounted for me:
[tdh@wont ~]> sudo mount kanigix:/ /mnt [tdh@wont ~]> cd /mnt cd export [tdh@wont /mnt]> cd export [tdh@wont export]> cd home [tdh@wont home]> ls -la total 4 drwxr-xr-x 15 root root 512 Mar 27 12:35 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 ..
If I go in via the automounter, what happens?
[tdh@wont home]> cd /net/kanigix/export/home ls -la [tdh@wont home]> ls -la total 44 drwxr-xr-x 15 root root 512 Mar 27 12:35 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 .. drwxr-xr-x 2 coach protos 512 Mar 27 12:35 coach drwxr-xr-x 2 haynest protos 512 Mar 27 12:35 haynest drwxr-xr-x 2 loghyr protos 512 Mar 27 12:35 loghyr drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 morgan protos 512 Mar 27 12:35 morgan drwxr-xr-x 2 mrx staff 512 Mar 27 12:35 mrx drwxr-xr-x 2 nfsv2 protos 512 Mar 20 01:31 nfsv2 drwxr-xr-x 2 nfsv3 protos 512 Mar 20 01:33 nfsv3 drwxr-xr-x 2 nfsv4 protos 512 Mar 20 01:01 nfsv4 drwxr-xr-x 2 spud protos 512 Mar 27 12:35 spud drwxr-xr-x 2 stacy protos 512 Mar 27 12:35 stacy drwxr-xr-x 4 tdh staff 512 Mar 27 00:21 tdh drwxr-xr-x 2 thomas staff 512 Mar 27 12:35 thomas
So the automounter has to be doing the work to mount these filesystems for us.
Lets now look at wont, which has a bunch of ZFS filesystems:
[tdh@wont home]> share - /export/zfs rw "" - /export/zfs/nfsv2 rw "" - /export/zfs/nfsv3 rw "" - /export/zfs/tdh rw "" - /zoo/isos/mnt/freebsd/d2 rw "" - /zoo/isos/mnt/Fedora/dvd ro "" - /export/zfs/nfsv4 rw "" - /export/zfs/spud rw "" - /export/zfs/haynest rw "" - /export/zfs/coach rw "" - /export/zfs/loghyr rw "" - /export/zfs/mrx rw "" - /export/zfs/thomas rw ""
And its /etc/dfs/dfstab is empty. What do we see with an explicit mount?
[tdh@kanigix ~]> sudo mkdir -p /nfsv4/wont [tdh@kanigix ~]> sudo mount -o intr wont:/ /nfsv4/wont [tdh@kanigix ~]> ls -la /nfsv4/wont total 9 drwxr-xr-x 42 root root 1024 Mar 23 11:21 . drwxr-xr-x 5 root root 512 Mar 27 14:34 .. drwxr-xr-x 4 root sys 512 Mar 21 15:28 export drwxr-xr-x 4 root sys 4 Mar 20 23:14 zoo
So NFSv4 has constructed a pseudo-fs entry for '/' which has the exported child nodes for export and zoo. This makes sense, under the NFSv4 spec, we need to be able to go from the root to each exported entry. If we look at the shares, /export and /zoo are the base prefixes.
What do we see if we enter these directories?
[tdh@kanigix ~]> cd /nfsv4/wont/zoo [tdh@kanigix zoo]> ls -la total 5 drwxr-xr-x 4 root sys 4 Mar 20 23:14 . drwxr-xr-x 42 root root 1024 Mar 23 11:21 ..
We still need to explicitly mount these exports if we want to view them. Remember, you can't automatically change filesystems from the server. Okay, lets play in /net.
[tdh@kanigix zoo]> cd /net/wont/zoo [tdh@kanigix zoo]> ls -al total 3 dr-xr-xr-x 2 root root 2 Mar 27 14:50 . dr-xr-xr-x 3 root root 3 Mar 27 14:32 .. dr-xr-xr-x 1 root root 1 Mar 27 14:50 isos
An interesting question is whether or not we are in a real filesystem? We really need to examine packets for signs of the psuedo-fs, but we can try some experiments:
[tdh@kanigix zoo]> mkdir foo mkdir: Failed to make directory "foo"; Operation not applicable
But we need to see what the local filesystem does to verify that the protocol implementation is doing something:
[tdh@wont home]> cd /zoo [tdh@wont /zoo]> ls -la total 12 drwxr-xr-x 4 root sys 4 Mar 20 23:14 . drwxr-xr-x 42 root root 1024 Mar 23 11:21 .. dr-xr-xr-x 3 root root 3 Mar 27 14:51 .zfs drwxr-xr-x 9 tdh sys 9 Mar 22 21:28 isos drwxr-xr-x 8 tdh staff 20 Mar 20 22:31 x86 [tdh@wont /zoo]> mkdir foo mkdir: Failed to make directory "foo"; Permission denied
I was pretty sure that this was ZFS protecting itself. Lets try some more:
[tdh@kanigix zoo]> cd isos [tdh@kanigix isos]> ls -la total 3 dr-xr-xr-x 2 root root 2 Mar 27 14:52 . dr-xr-xr-x 2 root root 2 Mar 27 14:50 .. dr-xr-xr-x 1 root root 1 Mar 27 14:52 mnt [tdh@kanigix isos]> mkdir foo mkdir: Failed to make directory "foo"; Operation not applicable
Local filesystem:
[tdh@wont /zoo]> cd isos/ [tdh@wont isos]> ls -la total 20 drwxr-xr-x 9 tdh sys 9 Mar 22 21:28 . drwxr-xr-x 4 root sys 4 Mar 20 23:14 .. dr-xr-xr-x 3 root root 3 Mar 27 14:52 .zfs drwxr-xr-x 3 tdh staff 3 Mar 22 17:01 Fedora drwxr-xr-x 2 tdh staff 4 Mar 21 21:29 FreeBSD drwxr-xr-x 3 tdh staff 3 Mar 22 11:35 bfus drwxr-xr-x 2 root root 3 Mar 21 16:16 dontcare drwxr-xr-x 4 tdh staff 4 Mar 22 21:30 mnt drwxr-xr-x 2 root root 4 Mar 21 16:28 sparc drwxr-xr-x 2 root root 6 Mar 21 20:57 x86 [tdh@wont isos]> mkdir foo
Okay, proof positive that we are indeed in the pseudo-fs. So the NFSv4 server implements a pseudo-fs which presents directory entries to the NSFv4 client. One thing it does is present a filehandle for the entry. That filehandle has to have marked in it that it is in the pseudo-fs. What happens if the admin goes ahead and exports that directory? And then the client passes in the pseudo-fs? Well the server has to be able to map that file handle to the new exported version.
[tdh@wont isos]> sudo share -F nfs -o rw=kanigix.internal.excfb.com -d "For kanigix only" /zoo/isos
To see some of my frustration with the share syntax, look at the related entry: [Open]Solaris and sharing subnets and single machines
Of course, client side caching can interfer:
[tdh@kanigix isos]> pwd /net/wont/zoo/isos [tdh@kanigix isos]> ls -la total 3 dr-xr-xr-x 2 root root 2 Mar 27 14:52 . dr-xr-xr-x 2 root root 2 Mar 27 14:50 .. dr-xr-xr-x 1 root root 1 Mar 27 14:52 mnt [tdh@kanigix isos]> cd .. [tdh@kanigix zoo]> ls -la total 3 dr-xr-xr-x 2 root root 2 Mar 27 14:50 . dr-xr-xr-x 3 root root 3 Mar 27 14:32 .. dr-xr-xr-x 2 root root 2 Mar 27 14:52 isos [tdh@kanigix zoo]> ls -al isos total 3 dr-xr-xr-x 2 root root 2 Mar 27 14:52 . dr-xr-xr-x 2 root root 2 Mar 27 14:50 .. dr-xr-xr-x 1 root root 1 Mar 27 14:52 mnt
After a client reboot:
[tdh@kanigix ~]> cd /net/wont/zoo/isos/ [tdh@kanigix isos]> ls -la total 20 drwxr-xr-x 10 tdh sys 10 Mar 27 14:52 . dr-xr-xr-x 2 root root 2 Mar 27 15:53 .. dr-xr-xr-x 3 root root 3 Mar 27 15:53 .zfs drwxr-xr-x 3 tdh staff 3 Mar 22 17:01 Fedora drwxr-xr-x 2 tdh staff 4 Mar 21 21:29 FreeBSD drwxr-xr-x 3 tdh staff 3 Mar 22 11:35 bfus drwxr-xr-x 2 root root 3 Mar 21 16:16 dontcare drwxr-xr-x 2 tdh staff 2 Mar 27 14:52 foo drwxr-xr-x 4 tdh staff 4 Mar 22 21:30 mnt drwxr-xr-x 2 root root 4 Mar 21 16:28 sparc drwxr-xr-x 2 root root 6 Mar 21 20:57 x86
While I forced the client cache to get nuked, it is quite possible for the client to have given that filehandle out to an application. That app could then present it back at a much later time.
There are other factors which impact the pseudo-fs - including security flavors. A simple introduction to some of the problems is in a presentation I gave at Connectathon 2004: mountd, exports, and the pseudo-fs in NFSv4 - Tom Haynes, Network Appliance. A key point here is that even though I did this presentation on the implementation of Data ONTAP, the same issues apply to every NFSv4 implementation. Data ONTAP has some quirks:
- If it isn't present, the default of '/vol/ROOT-NAME' is added.
- So, '/' typically means /vol/vol0.
- And thus '/' means something different to NFSv3 and NFSv4.
- Not implemented for NFSv4.
Technorati Tags: NFSv4 OpenSolaris ZFS
I wanted to make sandman only be on one subnet, my 192.168.3.0/24 one. I ended up doing a sys-unconfig to make sure I touched everything correctly. During the subsequent install, the DNS configuration failed to register. At this point, I was convinced it was the FreeBSD router/firewall. No evidence, just a gut feeling. Based on rules of thumb, it was the last thing to change and should be suspect.
I rebooted and I could see into the box from the 192.168.2.0/24 subnet:
[tdh@ultralord ~]> ping kanigix kanigix is alive [tdh@ultralord ~]> host kanigix kanigix.internal.excfb.com has address 192.168.3.109
kanigix and sandman are the same host.
And I could of course ping the router from kanigix:
# ping 192.168.3.1 192.168.3.1 is alive
But I couldn't ping the 192.168.2.1 router:
# ping 192.168.2.1 ^C#
Hmm, is 192.168.3.1 passing the packet on? Remember, this is where I think it is failing...
# traceroute 192.168.3.1 traceroute to 192.168.3.1 (192.168.3.1), 30 hops max, 40 byte packets 1 192.168.3.1 (192.168.3.1) 0.516 ms 0.412 ms 0.218 ms # traceroute 192.168.2.1 traceroute to 192.168.2.1 (192.168.2.1), 30 hops max, 40 byte packets 1 192.168.3.1 (192.168.3.1) 0.514 ms 0.398 ms 0.220 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 *^C
So, it appears that 192.168.3.1 is sitting on the packets.
The mistake here is that I am comparing apples to oranges. I can get in from ultralord, but I am not making sure I can get back out to ultralord.
What I should have done was find the IP of ultralord:
[tdh@ultralord ~]> host ultralord ultralord.internal.excfb.com has address 192.168.2.104
And then see if I could ping it:
# ping 192.168.2.104 192.168.2.104 is alive
I adtually ended up doing something like this, it just took 30 minutes longer than necessary. Anyway, once we figure out that 192.168.3.1 is passing packets, we need to explain the data. Well, 192.168.2.1 is not responding since it is a gateway appliance. It allows you to block WAN pings, but not LAN pings. These must be disabled by default.
The real question now is why is DNS hosed? I.e., before I thought the entire network system was horked, now I'm thinking just DNS.
# more resolv.conf domain internal.excfb.com nameserver 192.168.2.108 nameserver 192.168.2.1 # host adept Host adept not found: 5(REFUSED)
Sweet, that isn't a dropped packet. Is the nameserver misconfigured? I just added the 192.168.3.0/24 last week, and every host on that subnet also had a 192.168.2.0/24 connection. In the /etc/named.conf file on the nameserver, I had added the new subnet to the zones:
zone "2.168.192.in-addr.arpa" in {
type master;
file "internal.excfb.com/db.192.168.2";
allow-query {
192.168.2.0/24;
192.168.3.0/24;
};
allow-transfer {
192.168.2.0/24;
192.168.3.0/24;
};
};
But I failed to add it my "trusted-subnets" acl:
acl "trusted-subnets" {
192.168.2.0/24;
};
That messes up "match-clients" inside my view "internal":
view "internal" { // For the home network
match-clients {
localnets;
localhost;
"trusted-subnets";
};
So fix up the "trusted-subnets" acl:
acl "trusted-subnets" {
192.168.2.0/24;
192.168.3.0/24;
};
and restart the nameserver:
[root@adept named]# service named restart Stopping named: [ OK ] Starting named: [ OK ]
I still haven't ported my sendmail configuration to OpenSolaris... :->
Technorati Tags: network troubleshooting OpenSolaris
This is a companion piece to Linux firewall did not work, onto FreeBSD.
Remember that even NFSv4 does not mount across filesystems on OpenSolaris.
Consider the server with the following shares and filesystems:
[tdh@sandman home]> share - / rw "root" [tdh@sandman home]> df -h Filesystem size used avail capacity Mounted on /dev/dsk/c0t0d0s0 13G 4.8G 8.3G 37% / /devices 0K 0K 0K 0% /devices ctfs 0K 0K 0K 0% /system/contract proc 0K 0K 0K 0% /proc mnttab 0K 0K 0K 0% /etc/mnttab swap 1.2G 1.0M 1.2G 1% /etc/svc/volatile objfs 0K 0K 0K 0% /system/object fd 0K 0K 0K 0% /dev/fd swap 1.2G 16K 1.2G 1% /tmp swap 1.2G 56K 1.2G 1% /var/run /dev/dsk/c0t0d0s7 4.4G 4.6M 4.4G 1% /export/home
And consider a client with the following mounts from that client:
/nfsv4/sandman on sandman:/ remote/read/write/setuid/devices/xattr/dev=4c8000f on Mon Mar 27 00:09:25 2006 /nfsv4/kanigix on kanigix:/ remote/read/write/setuid/devices/xattr/dev=4c80011 on Mon Mar 27 00:10:46 2006
Okay, we want to do a test where we get a recursive directory listing of the home directories on the server:
[tdh@ultralord ~]> ls -alR /nfsv4/kanigix/export/ /nfsv4/kanigix/export/: total 6 drwxr-xr-x 3 root sys 512 Mar 18 00:27 . drwxr-xr-x 26 root root 512 Mar 20 01:34 .. drwxr-xr-x 2 root root 512 Mar 18 00:27 home /nfsv4/kanigix/export/home: ls: error reading directory /nfsv4/kanigix/export/home: Permission denied total 0 [tdh@ultralord ~]> sudo mount kanigix:/export/home /nfsv4/kanigix/export/home
And you still have to export the second filesystem:
[tdh@ultralord ~]> ls -alR /nfsv4/kanigix/export/ /nfsv4/kanigix/export/: total 6 drwxr-xr-x 3 root sys 512 Mar 18 00:27 . drwxr-xr-x 26 root root 512 Mar 20 01:34 .. drwxr-xr-x 2 root root 512 Mar 18 00:27 home /nfsv4/kanigix/export/home: ls: error reading directory /nfsv4/kanigix/export/home: Permission denied total 0 [tdh@ultralord ~]> cd /nfsv4/kanigix/export/home [tdh@ultralord home]> ls -la ls: error reading directory .: Permission denied total 0
And share it:
[tdh@sandman home]> sudo share -F nfs -o rw -d "homes" /export/home
Aaah!
[tdh@ultralord home]> ls -la ls: error reading directory .: Permission denied total 0
Hey, what is going on here? Client side caching!
[tdh@ultralord ~]> sudo mount sandman:/export/home /mnt [tdh@ultralord ~]> ls -la /mnt total 28 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 26 root root 512 Mar 24 19:04 .. drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 nobody nobody 512 Mar 20 01:31 nfsv2 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:33 nfsv3 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:01 nfsv4 drwxr-xr-x 2 tdh staff 512 Mar 26 23:51 tdh [tdh@ultralord ~]> sudo umount /mnt [tdh@ultralord ~]> sudo umount /nfsv4/sandman/export/home/ [tdh@ultralord ~]> sudo mount sandman:/export/home /nfsv4/sandman/export/home/ [tdh@ultralord ~]> ls -la /nfsv4/sandman/export/home/ total 4 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 .. [tdh@ultralord ~]> ls -la /nfsv4/sandman/export/home/ total 4 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 .. [tdh@ultralord home]> cd [tdh@ultralord ~]> sudo umount /nfsv4/kanigix/export/home [tdh@ultralord ~]> sudo mount kanigix:/export/home /nfsv4/kanigix/export/home [tdh@ultralord ~]> ls -alR /nfsv4/kanigix/export/ /nfsv4/kanigix/export/: total 6 drwxr-xr-x 3 root sys 512 Mar 18 00:27 . drwxr-xr-x 26 root root 512 Mar 20 01:34 .. drwxr-xr-x 7 root root 512 Mar 26 23:51 home /nfsv4/kanigix/export/home: total 4 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 ..
Okay, I'm getting mad and tired here. Pilot error? From scratch:
[tdh@ultralord ~]> sudo umount /mnt [tdh@ultralord ~]> sudo mount sandman:/ /mnt [tdh@ultralord ~]> sudo mount sandman:/export/home /mnt/export/home [tdh@ultralord ~]> ls -al /mnt/export/home total 28 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 .. drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 nobody nobody 512 Mar 20 01:31 nfsv2 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:33 nfsv3 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:01 nfsv4 drwxr-xr-x 2 tdh staff 512 Mar 26 23:51 tdh
I'm going to go with the permission denied being stuck on the top level mount:
[tdh@ultralord ~]> sudo umount /nfsv4/sandman/export/home/ [tdh@ultralord ~]> sudo umount /nfsv4/sandman [tdh@ultralord ~]> sudo mount sandman:/ /nfsv4/sandman [tdh@ultralord ~]> sudo mount sandman:/export/home /nfsv4/sandman/export/home/ [tdh@ultralord ~]> ls -la /nfsv4/sandman/export/home/ total 28 drwxr-xr-x 7 root root 512 Mar 26 23:51 . drwxr-xr-x 3 root sys 512 Mar 18 00:27 .. drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 nobody nobody 512 Mar 20 01:31 nfsv2 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:33 nfsv3 drwxr-xr-x 2 nobody nobody 512 Mar 20 01:01 nfsv4 drwxr-xr-x 2 tdh staff 512 Mar 26 23:51 tdh
All of this is going to be why ZFS will be a big administrative win with filesystem management.
Technorati Tags: NFSv4 OpenSolaris ZFS
Okay, the iptables support of nth and random is broken for modern kernels. I don't want to futz with a 2.4 kernel and I don't want to spend more days learning the iptables source.
Onto ipfw and FreeBSD 6.0! I'm using the default GENERIC kernel:
[tdh@mrx /etc]> uname -a FreeBSD mrx 6.0-RELEASE FreeBSD 6.0-RELEASE #0: Thu Nov 3 09:36:13 UTC 2005 root@x64.samsco.home:/usr/obj/usr/src/sys/GENERIC i386
And to enable the firewall, I add this to /etc/rc.conf:
firewall_enable="YES" firewall_type="open"
Actually, my entire file is:
# -- sysinstall generated deltas -- # Sat Mar 25 16:08:23 2006 # Created: Sat Mar 25 16:08:23 2006 # Enable network daemons for user convenience. # Please make all changes to this file, not to /etc/defaults/rc.conf. # This file now contains just the overrides from /etc/defaults/rc.conf. firewall_enable="YES" firewall_type="open" gateway_enable="YES" hostname="mrx" nfs_client_enable="YES" nfs_server_enable="YES" rpcbind_enable="YES" sshd_enable="YES" usbd_enable="YES" # -- sysinstall generated deltas -- # Sat Mar 25 22:52:02 2006 ifconfig_rl0="inet 192.168.2.101 netmask 255.255.255.0" ifconfig_fxp0="inet 192.168.3.1 netmask 255.255.255.0" defaultrouter="192.168.2.1" gdm_enable="yes"
I must admit that installing the FreeBSD ports is a pain, but configuring things in the kernel, the routes, etc, is much nicer than Linux. A lot of good examples are in the FreeBSD Handbook.
I also added some variables to /etc/sysctl.conf:
net.inet.ip.fw.verbose=1 net.inet.ip.fw.verbose_limit=5
Again, getting the FreeBSD firewall up and running was much easier for me than the Linux iptables. I must admit I've done the FreeBSD one in the past and I've also used it to simulate WAN delays in a testbed. (Thanks sr for really setting it up!).
Okay, when you boot up, you get the default firewall (default because of my options in /etc/rc.conf):
mrx# ipfw list 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 65000 allow ip from any to any 65535 deny ip from any to any
First we test it:
[tdh@mrx ~]> ping adept PING adept.internal.excfb.com (192.168.2.108): 56 data bytes 64 bytes from 192.168.2.108: icmp_seq=0 ttl=64 time=0.129 ms 64 bytes from 192.168.2.108: icmp_seq=1 ttl=64 time=0.189 ms ^C --- adept.internal.excfb.com ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.129/0.159/0.189/0.030 ms [tdh@mrx ~]> ping kanigix PING kanigix.internal.excfb.com (192.168.3.109): 56 data bytes 64 bytes from 192.168.3.109: icmp_seq=0 ttl=255 time=0.619 ms 64 bytes from 192.168.3.109: icmp_seq=1 ttl=255 time=0.235 ms ^C --- kanigix.internal.excfb.com ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.235/0.427/0.619/0.192 ms
And now we add in the rule to drop 10% of NFS traffic. Note that I don't care where the traffic is flowing - it could be a client request or a server reply. It could also be to the firewall. I just want to not think too hard and drop 10% on the floor:
mrx# ipfw add 7000 prob .1 deny all from any to any 2049 07000 prob 0.100000 deny ip from any to any dst-port 2049 mrx# ipfw list 00100 allow ip from any to any via lo0 00200 deny ip from any to 127.0.0.0/8 00300 deny ip from 127.0.0.0/8 to any 07000 prob 0.100000 deny ip from any to any dst-port 2049 65000 allow ip from any to any 65535 deny ip from any to any
My test system is setup as such:
mrx - FreeBSD 6.0 firewall/router (192.168.2.101) drx - "" (192.168.3.1) sandman - Nevada b35 Ultra 5 (192.168.2.109) kanigix - "" (192.168.3.109) ultralord - Nevada b35 Ultra 10 (192.168.2.104) wont - Nevada b35 Opteron Special (192.168.2.105)
I had earlier mounted both sandman:/ and kanigix:/ on ultralord.
Here is the output from ls -la on /nfsv4/sandman:
# cd /nfsv4/sandman # ls -la total 94 drwxr-xr-x 26 root root 512 Mar 20 01:34 . drwxr-xr-x 4 root root 512 Mar 24 19:04 .. drwx------ 2 root root 512 Mar 18 08:35 .gconf drwx------ 2 root root 512 Mar 18 08:36 .gconfd drwx------ 3 root root 512 Mar 18 08:28 .sunw lrwxrwxrwx 1 root root 9 Mar 18 00:28 bin -> ./usr/bin drwxr-xr-x 2 root nobody 512 Mar 18 19:54 cdrom drwxr-xr-x 18 root sys 4096 Mar 24 01:36 dev drwxr-xr-x 7 root sys 512 Mar 18 02:23 devices drwxr-xr-x 80 root sys 4608 Mar 24 19:07 etc drwxr-xr-x 3 root sys 512 Mar 18 00:27 export dr-xr-xr-x 2 root root 512 Mar 18 00:54 home drwxr-xr-x 14 root sys 512 Mar 18 01:47 kernel drwxr-xr-x 7 root bin 5632 Mar 18 02:01 lib drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 root sys 512 Mar 18 00:28 mnt dr-xr-xr-x 2 root root 512 Mar 18 08:27 net drwxr-xr-x 3 root root 512 Mar 20 01:34 nfsv4 drwxr-xr-x 6 root sys 512 Mar 18 08:38 opt drwxr-xr-x 38 root sys 1536 Mar 18 01:19 platform dr-xr-xr-x 2 root root 512 Mar 18 00:28 proc drwxr-xr-x 2 root sys 1024 Mar 18 01:22 sbin drwxr-xr-x 4 root root 512 Mar 18 00:28 system drwxrwxrwt 2 root sys 512 Mar 18 02:18 tmp drwxr-xr-x 37 root sys 1024 Mar 18 01:37 usr drwxr-xr-x 43 root sys 1024 Mar 18 08:28 var drwxr-xr-x 2 root root 512 Mar 18 08:28 vol
And here is the output from ls -la on /nfsv4/kanigix:
# cd /nfsv4/kanigix # ls -la ./proc: Operation not supported on transport endpoint ./home: Operation not supported on transport endpoint ./net: Operation not supported on transport endpoint total 88 drwxr-xr-x 26 root root 512 Mar 20 01:34 . drwxr-xr-x 4 root root 512 Mar 24 19:04 .. drwx------ 2 root root 512 Mar 18 08:35 .gconf drwx------ 2 root root 512 Mar 18 08:36 .gconfd drwx------ 3 root root 512 Mar 18 08:28 .sunw lrwxrwxrwx 1 root root 9 Mar 18 00:28 bin -> ./usr/bin drwxr-xr-x 2 root nobody 512 Mar 18 19:54 cdrom drwxr-xr-x 18 root sys 4096 Mar 24 01:36 dev drwxr-xr-x 7 root sys 512 Mar 18 02:23 devices drwxr-xr-x 80 root sys 4608 Mar 24 19:07 etc drwxr-xr-x 3 root sys 512 Mar 18 00:27 export drwxr-xr-x 14 root sys 512 Mar 18 01:47 kernel drwxr-xr-x 7 root bin 5632 Mar 18 02:01 lib drwx------ 2 root root 8192 Mar 18 00:27 lost+found drwxr-xr-x 2 root sys 512 Mar 18 00:28 mnt drwxr-xr-x 3 root root 512 Mar 20 01:34 nfsv4 drwxr-xr-x 6 root sys 512 Mar 18 08:38 opt drwxr-xr-x 38 root sys 1536 Mar 18 01:19 platform drwxr-xr-x 2 root sys 1024 Mar 18 01:22 sbin drwxr-xr-x 4 root root 512 Mar 18 00:28 system drwxrwxrwt 2 root sys 512 Mar 18 02:18 tmp drwxr-xr-x 37 root sys 1024 Mar 18 01:37 usr drwxr-xr-x 43 root sys 1024 Mar 18 08:28 var drwxr-xr-x 2 root root 512 Mar 18 08:28 vol
So, I'd want to get packet traces and investigate the "Operation not supported" messages to see if they are good messages to report.
But I have limited proof my test harness is indeed working.
Okay, to recap - random packet dropping is in place. I'll need to start looking at packet traces for differences and to see if behaviour is correct when a packet is dropped. One data point is that a recursive directory listing of the root is much slower with 10% packet loss.
Technorati Tags: NFSv4 OpenSolaris iptables ipfw FreeBSD