Hasham's blog

After Creating a Zone successfully on my Solaris 10 08/07 on  x64  system, I encountered the “useradd” issue. When I ran "useradd" in the zone resulted in error as shown below:

root@zone1:[/]# useradd -g staff -m -d /export/test -s /usr/bin/bash -c "Test User" test
UX: useradd: ERROR: Cannot update system files - login cannot be created.
root@zone1:[/]#

Zone creation/install process was without any error messages. So as first step, looked at the man page for useradd. The error is documented in the DIAGNOSTICS section of the man page.
The following indicates that the /etc/passwd or /etc/shadow files do not exist:
UX: useradd: ERROR: Cannot update system files - login cannot be created.

Good feeling of knowing the cause of the problem was short lived. These files were present on the system
root@zone1:[/]# ls -lt /etc/passwd
-rw-r--r--   1 root     sys          716 Jul  6 08:50 /etc/passwd
root@zone1:[/]# ls -lt /etc/shadow
-r--------   1 root     sys          372 Jul  6 08:50 /etc/shadow

Are these files corrupt or inconsistent? It was the next thing to verify. If these files edited manually(which is not a good practice), may get out of sync. I verified this by running “pwck”.  No errors reported, indication of files in order.

OK, so what next - May be DTrace ? - I had not used DTrace in Zones before, I thought this would be good problem to solve using DTrace. First target was to see if any useful scripts in "DTrace Toolkit" that could of help.

Zones by default, run with reduced set of privileges for security reasons. If you want, you can grant additional privileges to Zones. Not all privileges can be granted. See Solaris Container and Resource Manager Guide  for more details. Note the pros & cons of granting privileges to zones. Since I am the only user for this system, the decision was easy for me.

For running DTrace, you need to grant  "dtrace_user" and "dtrace_proc" privileges to the zone. See the steps below:

Without privileges this how the output of "Listing DTrace Probes"  looks
root@zone1:[/etc]dtrace -l
ID PROVIDER MODULE FUNCTION NAME

No probes are available as you can see from above output

Grating Privileges:
global# zonecfg -z mysql2 info limitpriv                 <====== List Current Privileges for the zone
limitpriv:

Add Privilges using the below commands:

global# zonecfg -z mysql2 set limitpriv=default,dtrace_proc,dtrace_user
global# zonecfg -z mysql2 info limitpriv                    <====== List modified Privilges for the zone
limitpriv: default,dtrace_proc,dtrace_user

Reboot the Zone to reflect the new settings

global# zoneadm -z zone1 reboot

root@zone1:[/]dtrace -l|wc -l
     483
root@zone1:[/]
Note that even after granting the DTrace privileges, LIMITED number of probes are available within a zone(Only a subset of what is available in global zone). This number is 40000 plus when you are in global zone.

DTrace Toolkit has lot useful scripts. For this problem, which appears to be "some files missing" , the toolkit script "errinfo" was very useful. It reported syscall failure.

root@zone1:[/DTraceToolkit-0.99]./errinfo -n useradd
            EXEC          SYSCALL  ERR  DESC
         useradd            xstat    2  No such file or directory
         useradd             open    2  No such file or directory
         useradd            xstat    2  No such file or directory
         useradd             open    2  No such file or directory
^C
root@zone1:[/DTraceToolkit-0.99]

As you can see from above output, open, xstat is failing when running "useradd". Next step was to find out for which file(s) these system calls are failing. I used another script from the toolkit "statsnoop"

root@zone1:[/DTraceToolkit-0.99]./statsnoop -t xstat
  UID    PID COMM          FD PATH
    0  15838 passmgmt       0 /usr/sbin/passmgmt
    0  15838 passmgmt       0 /lib/libsecdb.so.1
    0  15838 passmgmt       0 /lib/libc.so.1
    0  15838 passmgmt       0 /lib/libnsl.so.1
    0  15838 passmgmt       0 /lib/libcmd.so.1
    0  15838 passmgmt       0 /etc/passwd
    0  15838 passmgmt       0 /etc/shadow
   0  15838 passmgmt      -1 /etc/user_attr
    0  15837 useradd        0 /usr/sbin/useradd
    0  15837 useradd        0 /lib/libsecdb.so.1
    0  15837 useradd       -1 /lib/libproject.so.1
    0  15837 useradd        0 /usr/lib/libproject.so.1
    0  15837 useradd        0 /lib/libc.so.1
    0  15837 useradd        0 /lib/libnsl.so.1
    0  15837 useradd        0 /lib/libcmd.so.1
    0  15837 useradd        0 /lib/libproc.so.1
    0  15837 useradd        0 /lib/librtld_db.so.1
    0  15837 useradd        0 /lib/libelf.so.1
    0  15837 useradd        0 /lib/libctf.so.1

From the "statsnoop" script output, you can see that, it is returning an error for "/etc/user_attr" file. On verification, noticed that this file doesn't exist.

root@zone1:[/DTraceToolkit-0.99]ls -lt /etc/user_attr
/etc/user_attr: No such file or directory
root@zone1:[/DTraceToolkit-0.99]

I faced this problem right after the Zone installation. Given that, no other changes have been made to the zone after install, missing file from the zone could be because of incomplete install of a package. For finding out which package this file belongs to, I ran the following commands in the global zone where "useradd" was working fine.

global# pkgchk -l -p /etc/user_attr
Pathname: /etc/user_attr
Type: editted file
Expected mode: 0644
Expected owner: root
Expected group: sys
Referenced by the following packages:
        SUNWcsr        SUNWmconr      SUNWpostgr-82-server-data-root
Current status: installed

global# pkgchk -a SUNWcsr
global#
No errors reported. Package installation appears fine in the global zone. When I ran the same command in Zone, I got the error as reported below:

root@zone1:[/] pkgchk -a SUNWcsr

ERROR: /dev
    group name <sys> expected <root> actual
ERROR: /etc/user_attr
    pathname does not exist
root@zone1:[/]

By adding DTrace Specific Privileges to Zones, we were able to run DTrace in zones and identify the root cause of the problem. The file "/etc/user_attr" which is part of SUNWcsr package was missing. After restoring it, "useradd" command  works without any issues.