This blog entry is a continuation of my previous blog entry, Observing the Solaris Kernel.

For my second demo I'll take a look at the anatomy of a simple process. As before, let's start out with a really simple program:

% cat simple2.c
int
main(void)
{
        sleep(99999);
        return (0);
}
% gcc -o simple2 simple2.c
% ./simple2
^Z[1] + Stopped (SIGTSTP)        ./simple2
%

In another window, as root, I fire up mdb in kernel target mode, and look up the proc structure of the process:

root@rutamaya > mdb -k
Loading modules: [ unix krtld genunix specfs dtrace pcplusmp ufs ip sctp usba fctl nca lofs random nfs logindmux ptm cpc fcip sppp ]
mdb> ::ps !head -1
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
mdb> ::ps !grep simple2
R 127247 127218 127247 127193 125946 0x42004000 fffffead763de3c0 simple2
mdb> fffffead763de3c0::print proc_t
{
    p_exec = 0xfffffe819091a840
    p_as = 0xffffffff899c1010
    p_lockp = 0xffffffff84a6db40
    p_crlock = {
        _opaque = [ 0 ]
    }
    p_cred = 0xffffffff923e14e8
...

Let's take a look at the context of the process in the kernel, to see what it looks like after I sent it a SIGSTOP using ctrl+z from the terminal:

mdb> fffffead763de3c0::walk thread
fffffe814473abc0
mdb> fffffe814473abc0::findstack -v
stack pointer for thread fffffe814473abc0: fffffe80007abb50
[ fffffe80007abb50 _resume_from_idle+0xde() ]
  fffffe80007abb90 swtch+0x241()
  fffffe80007abc00 stop+0xa68(5, 18)
  fffffe80007abc40 isjobstop+0xd7(18)
  fffffe80007abcf0 issig_forreal+0x48c()
  fffffe80007abd20 issig+0x28(0)
  fffffe80007abda0 cv_timedwait_sig+0x266(fffffe814473ad96, fffffe814473ad98,
  73de75e)
  fffffe80007abe20 cv_waituntil_sig+0xab(fffffe814473ad96, fffffe814473ad98,
  fffffe80007abe70, 1)
  fffffe80007abeb0 nanosleep+0x141(8047cb0, 8047cb8)
  fffffe80007abf00 sys_syscall32+0x1ff()

::walk thread takes a process structure pointer and "walks" all of the threads in the process (in Solaris there is a 1:1 mapping of user thread to kernel thread these days). Since our simple program is single-threaded this yields a single thread pointer, which we could have also gotten at by hand by looking at the p_tlist member of the proc structure:

mdb> fffffead763de3c0::print proc_t p_tlist
p_tlist = 0xfffffe814473abc0

::findstack takes a thread pointer and walks the thread stack in the kernel. We can see that cv_waituntil_sig() was called from nanosleep(), and it was interrupted by the SIGSTOP signal causing the process to call stop().

Using the pipe facility in mdb and a new dcmd I haven't yet used, ::pid2proc, I can also shorten everything I've done so far into an equivalent two-liner:

mdb> ::ps !grep simple2
R 127247 127218 127247 127193 125946 0x42004000 fffffead763de3c0 simple2
mdb> 0t127247::pid2proc | ::walk thread | ::findstack -v
stack pointer for thread fffffe814473abc0: fffffe80007abb50
[ fffffe80007abb50 _resume_from_idle+0xde() ]
  fffffe80007abb90 swtch+0x241()
  fffffe80007abc00 stop+0xa68(5, 18)
  fffffe80007abc40 isjobstop+0xd7(18)
  fffffe80007abcf0 issig_forreal+0x48c()
  fffffe80007abd20 issig+0x28(0)
  fffffe80007abda0 cv_timedwait_sig+0x266(fffffe814473ad96, fffffe814473ad98,
  73de75e)
  fffffe80007abe20 cv_waituntil_sig+0xab(fffffe814473ad96, fffffe814473ad98,
  fffffe80007abe70, 1)
  fffffe80007abeb0 nanosleep+0x141(8047cb0, 8047cb8)
  fffffe80007abf00 sys_syscall32+0x1ff()

Note the 0t prefix is needed because the PID is displayed by ::ps in decimal, just as if I had run the ps command in a terminal. In fact I could have done this from inside mdb using the shell pipe facility I've already used in this demo:

mdb> !ps -ef | grep simple2
   elowe 127247 127218   0 10:04:24 pts/1       0:00 ./simple2

In my first demo, I examined the user and group ID within a file; as we saw from running ::print to look at the proc structure, p_cred also contains a pointer to a cred structure, which holds the identity of a unique user on the system. Since I started the process from another terminal window under my normal UNIX account, the cred structure should contain the credentials associated with my UNIX login.

mdb> fffffead7ceb43c0::print proc_t p_cred|::print cred_t
{
    cr_ref = 0x32
    cr_uid = 0x1ebfa
    cr_gid = 0xa
...
mdb> fffffead7ceb43c0::print proc_t p_cred|::print cred_t cr_uid|::map =D
                125946

That's me. Since ::print likes hex more than I do, I used the ::map dcmd to pipe the dot from ::print cred_t cr_uid into =D, which is the mdb syntax to display a number in integer decimal format.

There are plenty of interesting things to explore on your own just by starting from the process structure, so this is a great start for folks who are just starting out learning about UNIX internals.

Technorati Tags: [OpenSolaris Solaris]