Sherlock Holmes and The Adventure of the Odd Permissions

"Come, Watson, come!" he cried. "The game is afoot. Not a word! Into your clothes and come!"

Well, it wasn't quite as dramatic as that, and it wasn't a "three pipe problem", but a little while ago I was handed a puzzler with an unexpected result with ZFS file systems. Specifically, a non-root user created a directory on ZFS, put a file and a subdirectory in it, and then was unable to remove it. Doing the same with UFS worked as expected. This violates the Principle of Least Astonishment!

tank/fs1> mkdir temp		create a directory	
tank/fs1> cd temp
tank/fs1/temp> touch a		put a file in it
tank/fs1/temp> mkdir bak	and a subdirectory
tank/fs1/temp> cd ..
tank/fs1> ls -lad temp		yup, there it is
drwxr-xr-x   3 joeuser99 smile          4 Apr 15 15:46 temp

tank/fs1> rm -rf temp		
tank/fs1> ls -lad temp		huh? why is it still there
drwxr-xr-x   3 joeuser99 smile          3 Apr 15 15:46 temp

tank/fs1> cd temp		let's get closer
tank/fs1/temp> ls -la		the file is gone, but subdir remains
total 9
drwxr-xr-x   3 joeuser99 smile          3 Apr 15 15:46 .
drwxr-x---   4 joeuser99 smile          9 Apr 15 15:46 ..
drwxr-xr-x   2 joeuser99 smile          2 Apr 15 15:46 bak
tank/fs1/temp> cd ..
tank/fs1> chmod -R 777 temp	brute force is always fun	
tank/fs1> ls -ald temp
drwxrwxrwx   3 joeuser99 smile          3 Apr 15 15:46 temp
tank/fs1> rm -rf temp		remove it
tank/fs1> ls -lad temp		it doesn't want to be removed
drwxrwxrwx   3 joeuser99 smile          3 Apr 15 15:46 temp

tank/fs1> cd temp
tank/fs1/temp> ls -la  		Blimey! Same as before
total 9
drwxrwxrwx   3 joeuser99 smile          3 Apr 15 15:46 .
drwxr-x---   4 joeuser99 smile          9 Apr 15 15:46 ..
drwxrwxrwx   2 joeuser99 smile          2 Apr 15 15:46 bak

Well, that makes no sense - if you create a directory or file, you should be able to remove it, right? Not necessarily!

ZFS works differently from traditional UFS: ZFS uses a pure ACL model, unlike UFS which either has ACL settings or permission bits. If you're used to traditional Unix file permissions (nicely described here and here, as well as hundreds of other places) that's how it works, but when you're using Access Control Lists the permission to write to a directory doesn't necessarily imply the permission to remove objects placed in it.

Specifically, there are separate access privileges: add_file (permission to add a new file to a directory), add_subdirectory (permission to create a subdirectory), delete (permission to delete a file), and delete_child (permission to delete a file or directory within a directory)

"Data! Data! Data!" he cried impatiently. "I can't make bricks without clay."

In the user's situation, things worked as expected when the ZFS pool had the default settings of:

  aclmode         groupmask           default
  aclinherit      restricted          default

but not when it had
  aclmode         passthrough         local
  aclinherit      passthrough         local

So, the question here is: what were the ACL settings and permissions on the parent filesystem? Unfortunately, the data was removed by brute force so I never got the settings that caused the unexpected results, and I wasn't able to duplicate this using default privileges and ACL settings with either combination of aclmode and aclinherit. I imagine that somewhere there were non-default ACL settings that specifically granted the add_* permissions without the corresponding delete_* permissions, but I can't know for sure without data.

Where to get more data

Clearly, changing the default ACL settings and how they are passed via inheritance can have surprising effects. For a very good discussion on how ACLs work with ZFS see Mark Shellenbaum's 2005 blog ZFS ACLs which has a clear explanation and useful examples to demonstrate how ACLs work. For the reference document, see Chapter 8 Using ACLs to Protect ZFS Files in the Solaris ZFS Administration Guide

Apologies to Arthur Conan Doyle for lifting Sherlock Holmes quotes from The Red-Headed League, The Adventure of the Abbey Grange, and The Adventure of the Copper Beeches.

Posted by jsavit [Sun] ( May 11, 2009 10:08 AM ) Permalink