So I decided to learn python - why? because it is used by Mercurial. And there is at least one of the gatekeeping scripts which I needed to hack for the nfs41-gate.

I bought Learning Python, Third Edition by Mark Lutz because the local Borders did not have Programming Python, Third Edition. From some reviews I read, it would probably have been a better fit for me.

I know I can find most of what I want on the net, but I wanted a printed resource.

Anyway, I had a question right off the bat - about whether if file a imports modules b and c, what happens if c also imports b? Deeper in the book that I've read, it does state that an import is equivalent to load the file if it is not already loaded. But that doesn't help me learn the language. :->

The following example is quite simple, but effective in answering the question for me:

a.py

> cat a.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing b from a"
import b

print "importing c from a"
import c

b.py

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

c.py

> cat c.py
#!/usr/bin/python

import b

title = "This is the file c.py!"

print title

Test 1

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
importing c from a
This is the file c.py!

So we see that if a has loaded it, then c will not. How about the other way?

a2.py

> cat a2.py
#!/usr/bin/python

title = "This is the file a.py!"

print title

print "importing c from a"
import c

print "importing b from a"
import b

print "and b's title is"
print b.title

Test 2

> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

We see that c loads b and that b's attributes are visible from a.

What would really help me here is if b could state the call stack of what is importing it.

Test 3

A simple change fails:

> cat b.py
#!/usr/bin/python

title = "This is the file b.py!"

print title

print "Called from", __file__

but it does show the effect of byte code compilation:

> ./a.py
This is the file a.py!
importing b from a
This is the file b.py!
Called from /home/tdh/python/b.py
importing c from a
This is the file c.py!
> ./a2.py
This is the file a.py!
importing c from a
This is the file b.py!
Called from /home/tdh/python/b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!

I can see the "nesting" if I pop into an interactive session:

> python
>>> import a2
This is the file a.py!
importing c from a
This is the file b.py!
Called from b.pyc
This is the file c.py!
importing b from a
and b's title is
This is the file b.py!
>>> a2.__dict__.keys()
['c', 'b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.__dict__.keys()
['b', 'title', '__builtins__', '__file__', '__name__', '__doc__']
>>> a2.c.b.__dict__.keys()
['__builtins__', '__name__', '__file__', '__doc__', 'title']

But this doesn't answer my question of how to figure this out recursively. I.e., I guess I am looking for a parent "pointer" and I could walk it to get my answer.

But I've still learned more than just reading the book linearly.

AUTH_SYS is an insecure security mode, yet it is commonly used within companies. It can be used as the proverbial open lock on a door - the fact that the lock is there means do not enter. But I've seen people terminated for ignoring that lock.

With that in mind, I want to go over the simple security schemes employed within a company and show why they don't work. The punchline will be of course Kerberos. Speaking of myths, one is that you need NFSv4 in order to deploy Kerberos. You don't - common servers and clients easily speak Kerberos with NFSv3. And ignore NFSv2, please, please.

Export Security

With an export (or share), the most lax security is typically the default:

[root@pnfs-9-26 ~]> zfs create rootpool/export/home/secure
[root@pnfs-9-26 ~]> share
[root@pnfs-9-26 ~]> zfs set sharenfs=on rootpool/export/home/secure
[root@pnfs-9-26 ~]> share
-@rootpool/exp  /export/home/secure   rw   ""  

I.e., every machine in the world can mount pnfs-9-26:/export/home/secure. The reasons for this default are simple:

1. When the model was first developed
   1. the number of exports and clients was limited
   2. the degree of interconnectedness outside of the organization was limited
2. It is hard to know beforehand where to limit access.

By default, root has access almost like any other user, but it is mapped to the user nobody. We can see this here if we grant wide open permissions on the export:

[root@pnfs-9-26 ~]> chmod 777 /export/home/secure
[root@pnfs-9-26 ~]> ls -la /export/home/secure
total 6
drwxrwxrwx   2 root     root           2 Oct  5 11:39 .
drwxr-xr-x   5 th199096 staff          6 Oct  5 11:39 ..

We should be able to create a file as anyone from another machine:

[root@jhereg ~]> mount -o vers=3 pnfs-9-26:/export/home/secure /mnt
[root@jhereg ~]> touch /mnt/i_am_root

That worked:

[root@pnfs-9-26 secure]> ls -la
total 7
drwxrwxrwx   2 root     root           3 Oct  5 11:51 .
drwxr-xr-x   5 th199096 staff          6 Oct  5 11:39 ..
-rw-r--r--   1 nobody   nobody         0 Oct  5 11:51 i_am_root

Notice that root has been mapped to nobody. What happens if we do it as a normal user:

[th199096@jhereg ~]> touch /mnt/i_am_jhereg
[th199096@jhereg ~]> touch /mnt/i_am_th199096

And we get the correct user:

[root@pnfs-9-26 secure]> ls -la
total 9
drwxrwxrwx   2 root     root           5 Oct  5 11:54 .
drwxr-xr-x   5 th199096 staff          6 Oct  5 11:39 ..
-rw-r--r--   1 th199096 staff          0 Oct  5 11:54 i_am_jhereg
-rw-r--r--   1 nobody   nobody         0 Oct  5 11:51 i_am_root
-rw-r--r--   1 th199096 staff          0 Oct  5 11:54 i_am_th199096

Now what happens if we try to remove i_am_th199096 as root?

[root@jhereg ~]> rm /mnt/i_am_th199096
rm: /mnt/i_am_th199096: override protection 644 (yes/no)? y

anon

We are allowed to do that, but is it a property of being root or the permissions? We can check this with a simple change of the share:

[root@pnfs-9-26 secure]> zfs set sharenfs=anon=-1 rootpool/export/home/secure
[root@pnfs-9-26 secure]> share
-@rootpool/exp  /export/home/secure   anon=-1   ""  

See share_nfs(1M) for a description of anon. Notice I didn't specify whether rw is set or not. We can retry the delete:

[root@jhereg ~]> rm /mnt/i_am_jhereg
NFS3 getattr failed for pnfs-9-26: RPC: Authentication error; s1 = 13, s2 = 0
rm: /mnt/i_am_jhereg: Permission denied

If you want to make sure to deny root level access to a share, then you need to set anon=-1.

Conversely, if you want to enable root level access to a share, you can set anon=0:

[root@pnfs-9-26 secure]> zfs set sharenfs=anon=0 rootpool/export/home/secure
[root@pnfs-9-26 secure]> share
-@rootpool/exp  /export/home/secure   anon=0   ""  

I've recreated the two files in the background (which shows by the way that rw is the default). And when we test the deletion:

[root@jhereg ~]> rm /mnt/i_am_jhereg
[root@jhereg ~]> 

No pesky question that implies I am not a god!

root=

If I want to allow root access from one host but deny it from all others, I can use the root= access list:

[root@pnfs-9-26 secure]> zfs set sharenfs=root=pnfs-9-25.central.sun.com rootpool/export/home/secure
[root@pnfs-9-26 secure]> share
-@rootpool/exp  /export/home/secure   sec=sys,root=pnfs-9-25   ""  

PS: The sec=sys is stating this is an AUTH_SYS share. Also, since I am using DNS for hosts in /etc/resolv.conf, I need a FQDN.

Try to remove:

[root@jhereg ~]> rm /mnt/i_am_th199096 
rm: /mnt/i_am_th199096: override protection 644 (yes/no)? yes

Since it worked and we got a prompt, it has to be the permission set which is enabling this. If we tighten things down a bit more:

[root@pnfs-9-26 secure]> zfs set sharenfs=root=pnfs-9-25.central.sun.com,anon=-1 rootpool/export/home/secure
[root@pnfs-9-26 secure]> share
-@rootpool/exp  /export/home/secure   anon=-1,sec=sys,root=pnfs-9-25   ""  

We can see we are locked out:

[root@jhereg ~]> rm /mnt/i_am_root
rm: /mnt/i_am_root: Permission denied

versus

[root@pnfs-9-25 ~]> rm /mnt/i_am_root
[root@pnfs-9-25 ~]> 

And yet the other machine reigns supreme:

We'll revisit the use effectiveness of root= without anon=, when we look at permissions.

rw=

So we can keep machines from getting access altogether by restricting the rw= access list:

[root@pnfs-9-26 ~]> zfs set sharenfs=rw=pnfs-9-25.central.sun.com rootpool/export/home/secure
[root@pnfs-9-26 ~]> share
-@rootpool/exp  /export/home/secure   sec=sys,rw=pnfs-9-25.central.sun.com   ""  

which yields on the two clients:

[root@jhereg ~]> ls -la /mnt
/mnt: Permission denied

and

[root@pnfs-9-25 ~]> ls -la /mnt
drwxrwxrwx   2 root     root           6 Oct  5 19:33 .
drwxr-xr-x  36 root     root          39 Oct  5 19:11 ..
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_here
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs-9-25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs_9_25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_th199096

Note that the client jhereg must be caching a file handle for the root of the export /export/home/secure on the server pnfs-9-26. If it were not, we would have to reissue the mount request, which would have to fail. Also note, it is not just the mountd requests which have to check access list permissions. If it were, then the above operations would always work. SunOS used to work this way and the Solaris NFS team made a change back in the 1995/96 time frame, see for example Brent Callaghan's presentation at the 1996 Connectathon: NFS Client Authentication. And quickly, the security reason for doing so is the implication that if a rogue client someone sniffed out a valid file handle, then it had complete access to all of the information on that share.

ro=

We can likewise grant read only access via the ro= access list.

Access list interactions

All of rw, rw=, ro, and ro= interact as described by sharenfs(1M).

File Permissions

So access lists work on machines. If a machine is able to mount a share from a server, then all users on that client can access everything on that server. Right?

Wrong. The directory and file permissions determine user access. Contrast this with a model derived from a client only having one user logged in at a time. In that situation, it may not be the machine which is important but rather the user..

If I wanted to only grant access to a single user, then I would set the owner of the share to be that user and I would also set the permissions to be 700:

[root@pnfs-9-26 ~]> chown th199096:staff /export/home/secure/
[root@pnfs-9-26 ~]> chmod 700 /export/home/secure/
[root@pnfs-9-26 ~]> ls -la /export/home/secure/
total 10
drwx------   2 th199096 staff          6 Oct  5 19:33 .
drwxr-xr-x   5 th199096 staff          6 Oct  5 11:39 ..
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_here
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs-9-25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs_9_25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_th199096

And lets change the share to be wide open:

[root@pnfs-9-26 ~]> zfs set sharenfs=on rootpool/export/home/secure
[root@pnfs-9-26 ~]> share
-@rootpool/exp  /export/home/secure   rw   ""  

We see root access is denied (because it maps to nobody):

[root@pnfs-9-25 ~]> ls -la /mnt
/mnt: Permission denied
total 3

But on that same machine, th199096 is granted access:

[root@pnfs-9-25 ~]> su - th199096
[th199096@pnfs-9-25 ~]> ls -la /mnt
total 12
drwx------   2 th199096 staff          6 Oct  5 19:33 .
drwxr-xr-x  36 root     root          39 Oct  5 19:11 ..
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_here
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs-9-25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs_9_25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_th199096

By the way, if we grant either root= or anon=0 access, then this all goes out the window:

[root@pnfs-9-26 ~]> zfs set sharenfs=rw,anon=0 rootpool/export/home/secure

yields:

[root@pnfs-9-25 ~]> ls -la /mnt
total 12
drwx------   2 th199096 staff          6 Oct  5 19:33 .
drwxr-xr-x  36 root     root          39 Oct  5 19:11 ..
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_here
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs-9-25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:27 i_am_pnfs_9_25
-rw-r--r--   1 th199096 staff          0 Oct  5 13:30 i_am_th199096

A client's root only gets to boss things around if the server grants permission.

The final myth of AUTH_SYS

Take a server for which the root account is locked down. Assume admins who don't want an inadvertent 'rm -rf /net' to nuke their server, so by default they create shares of the form:

[root@pnfs-9-26 ~]> zfs set sharenfs=rw,anon=-1 rootpool/export/home/secure

And further, at some point someone decides to lock down a share's permissions, i.e., 700 on the user th199096.

How long would it take someone to get access over AUTH_SYS?

Not long - even though we know root access is out and we can assume they do not know my password. Since we use NIS, they can do a 'ypcat passwd | grep th199096' and grab my uid. Then they only have to create a dummy account a test machine.

What if we create a special account, not in NIS? Well, they may not have root access on the server, but if they have any access, then they could cd to the parent directory, issue an 'ls -la', see the user name, and then grep for it out of /etc/passwd.

You could lock down the machine, lock down the NIS database, etc. But the fact remains that if I can mount it, then I can create a simple script to try every UID until I get access. How many servers out there check for getattr storms?

The answer is to further restrict the access lists. But eventually, if I'm able to gain access to one of the restricted machines or if I can bring up my box with the same IP as one of the restricted machines, I can get access.

Kerberos

But all I need to do to combat this without all of these "extreme" measures is to enable Kerberos on the server:

[root@pnfs-9-26 ~]> zfs set sharenfs=sec=krb5,rw,anon=-1 rootpool/export/home/secure
[root@pnfs-9-26 ~]> share
-@rootpool/exp  /export/home/secure   anon=-1,sec=krb5,rw   ""  

I am the right user (actually my uid on pnfs-9-25 matches that of the uid of the user th199096 on pnfs-9-26), but it fails:

[th199096@pnfs-9-25 ~]> ls -al /mnt
NFS3 access failed for pnfs-9-26: RPC: Authentication error; s1 = 13, s2 = 0
/mnt: Permission denied
total 3

I call a "share" an "export" because I learned the terminology at another company, one based on the SunOS style and not the Solaris style. It turns out I have other expectations on how shares work. I thought the following was legal:

[root@pnfs-9-24 ~]> zfs set sharenfs=rw=pnfs-9-25:jhereg rootpool/export/home/secure

And all I got was:

[root@pnfs-9-25 ~]> mount pnfs-9-24:/export/home/secure /mnt
nfs mount: mount: /mnt: Permission denied

I reinstrumented mountd to spit out some debug messages and I saw:

[root@pnfs-9-24 ~]> Oct  5 16:04:27 pnfs-9-24 mountd[1598]: Considering |pnfs-9-25| vs |pnfs-9-25.Central.Sun.COM|
Oct  5 16:04:27 pnfs-9-24 mountd[1598]: Considering |jhereg| vs |pnfs-9-25.Central.Sun.COM|
Oct  5 16:04:27 pnfs-9-24 mountd[1598]: Considering |pnfs-9-25| vs |pnfs-9-25.Central.Sun.COM|
Oct  5 16:04:27 pnfs-9-24 mountd[1598]: Considering |jhereg| vs |pnfs-9-25.Central.Sun.COM|
Oct  5 16:04:27 pnfs-9-24 mountd[1598]: pnfs-9-25.Central.Sun.COM denied access to /export/home/secure

So it never considers the FQDN. Interesting, so what happens if we add it?

[root@pnfs-9-24 ~]> zfs set sharenfs=root=pnfs-9-25.Central.sun.com,anon=-1 rootpool/export/home/secure

We see:

[root@pnfs-9-25 ~]> mount pnfs-9-24:/export/home/secure /mnt
[root@pnfs-9-25 ~]> 

And on the console:

[root@pnfs-9-24 ~]> Oct  5 16:06:27 pnfs-9-24 mountd[1598]: Considering |pnfs-9-25.Central.sun.com| vs |pnfs-9-25.Central.Sun.COM|

By the way, the compare is case insensitive. This took me way longer to track down than I liked. And it had me going down dead-ends with other "bugs".

The share_nfs(1M) has this to say:

access_list The access_list argument is a colon-separated list whose components may be any number of the following: hostname The name of a host. With a server con- figured for DNS or LDAP naming in the nsswitch "hosts" entry, any hostname must be represented as a fully quali- fied DNS or LDAP name.

And sure enough:

[root@pnfs-9-24 ~]> grep hosts /etc/nsswitch.conf
# "hosts:" and "services:" in this file are used only if the
#hosts:      nis [NOTFOUND=return] files
hosts: files dns
# before searching the hosts databases.

Besides RTFMing myself, which I had done earlier, but not well enough, I was struck by the thought that I wish we had made this choice at a previous company. It solves a lot of problems, reduces a lot of name server queries (which was many of the problems), but is not as flexible. Consider a multi-homed client thorton which can either be thorton.central.sun.com or thorton.be.central.sun.com. With just rw=thorton, we can leverage the search domains to allow access to both interfaces as once.

But, depending on the ordering in the search domains, we may end up sending more name lookups than we want. Also, I've heard some sysadmins expose the belief that those interfaces represent different machines. And if you want both to have access, you explicitly grant them both access.