Weblog

System-wide AppCrash for special processes

As discussed in Easy way to install AppCrash system-wide, AppCrash can be installed as a daemon watching the entire system for crashing processes. This has proven to be the most practical way to use it. Most AppCrash users I know don't modify any of those scripts. They just use the ones supplied with the tool.

(Generally, I've noticed that ready-to-use tools performing specific tasks tend to be used much more than toolkits requiring the users to spend a lot of time and effort to learn and then to build their own tools.)

However, recently I've got a few reports from users of AppCrash saying pstack(1) and other such commands didn't work due to permission problems of the crashing process in some cases, particularly when the crashing processes are more than "user applications."

The original AppCrash design was that app_crash.d or app_crash_global.d executed the runme_on_app_crash script as the user who owned the crashing process. As it turned out, there is a problem with this approach in some cases: certain special processes have complex permission settings that prevent this design from working correctly.

Here are two examples:

• The Apache Web Server process crashed but pstack(1) didn't work. The reason was the Apache process was owned by "nobody" which is a special account with very few permissions. It is not even allowed to run pstack(1) on its own processes.

• The Xsun (X-server) process crashed but pstack(1) didn't work. Even though the X-server process is owned by the user who started the window system (CDE or GNOME), it has special permissions not allowing that user to run pstack(1) on it (among other things):

  % /usr/ucb/ps auxww | grep Xsun
  gregns   17077  3.9 22.1287688111848 ?        S   Jul 28 513:48 /usr/openwin/bin/Xsun :0 -defdepth 24 -nobanner -auth /var/dt/A:0-D0aigb
  % whoami
  gregns
  % pstack 17077
  pstack: cannot examine 17077: permission denied 
  

To resolve this problem, I've modified (simplified, actually) the system-wide AppCrash setup described in Easy way to install AppCrash system-wide to run the /opt/app_crash/runme_on_app_crash script as root, regardless of who owns the crashing process.

Morgan Herrington and I have discussed the implications of this design change.

On one hand, running the runme_on_app_crash script as root resolves the problem described above: pstack(1) and other such commands produce their output as required for any crashing process. On the other hand, it may create some problems.

Here are the issues we have considered.

• The main reason we used "su" in the DTrace script was that originally AppCrash was designed to enable users to catch crashes in their applications. Therefore it made sense to run the runme_on_app_crash script as the user who owned the crashing application. Also, even when AppCrash was installed by root, we wanted to keep the original implementation from granting root access to users. In the system-wide usage mode we are discussing here, this is not a problem since the runme_on_app_crash script in installed in the system area such as /opt/app_crash/runme_on_app_crash that requires root access anyway.

• The "root" account has *fewer* capabilities with regards to NFS than the user accounts. If a script running as root tries to write into an NFS-mounted file, it will have its user ID set to "nobody" (whose uid is -2), and thus will *not* be able to write into an NFS-mounted directory owned by a non-root user. In this case, however, script runme_on_app_crash script only writes to /var/tmp which is normally not NFS-mounted.

• Generally, it's best to do as few things as possible as "root," to prevent small mistakes from causing big problems. In this case, however, the chances of this script doing harm are very small. It only runs query commands such as pstack(1).

• Cleaning up /var/tmp may become a problem if you don't have root access. /var/tmp is a publicly writable directory with a sticky bit set:

  %  ls -ld /var/tmp
  drwxrwxrwt  12 root     sys        12288 Sep  1 15:42 /var/tmp/
  

  Therefore, files created by root can't be deleted from there by non-root:

  % cd /var/tmp
  % ls -l junk
  -rw-r--r--   1 root     root           0 Sep  1 15:39 junk
  % whoami
  gregns
  % rm junk
  rm: junk: override protection 644 (yes/no)? yes
  rm: junk not removed: Permission denied
  

  To resolve this problem, I've added the following statement at the end of the runme_on_app_crash script:

  chmod 666 $APPCRASH_OUT 
  

  This way, the resulting output file will still be owned by root, but anyone will be able to delete it at cleanup time:

  % ls -l junk
  -rw-rw-rw-   1 root     root           0 Sep  1 15:39 junk
  % whoami
  gregns
  % rm junk
  % 
  

• Running the runme_on_app_crash as root may change the environment settings. In this case, however, we are not using any environment variables.

• If an intruder manages to subvert the /opt/app_crash/runme_on_app_crash script to do something evil, there may be big trouble when that script is executed by root. In this case, however, it's not a danger. /opt has the same permissions as /. If an intruder has access to /opt, security is gone anyway.

The updated tarball with AppCrash is in the same place as before: app_crash.tar.gz. The installation instructions are here: Easy way to install AppCrash system-wide.

If you already have system-wide AppCrash installed, here's how you can update it:

• Update scripts /opt/app_crash/app_crash_global.d and /opt/app_crash/runme_on_app_crash from the latest app_crash.tar.gz.

• Then, as root, restart the AppCrash daemon by running the following commands:

  # /etc/init.d/app_crash stop
  # /etc/init.d/app_crash start
  

( Sep 06 2006, 11:11:36 AM EDT ) Permalink Comments [1]