greetings from network.com: network.com activities

The solaris pstack command can be used to view most user process stacks, but I recently encountered a situation where even the pstack -F could not be used to view the process stack.

I issued a zfs recv operation that never returned.

# ps -ef | grep zfs

    root  7485  7484   0 01:40:01 ?           0:00 zfs recv -F storage ...

# pstack -F 7485

[no output]

pstack could not grab control the process to display the stack.

My next option is to use mdb to view the stack - no quite as convenient as pstack, but more powerful ... here are the steps:

# mdb -k

/* get handle to the the zfs process using ::pgrep */

>::pgrep zfs

S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   7485   7484   7484   7484      0 0x4a004000 ffffff278437ca88 zfs

/* use the returned ADDR value to get the process threadlist */

>  ffffff278437ca88::threadlist
            ADDR             PROC              LWP CMD/LWPID
ffffff278437ca88 ffffff21da3b68e0                0 /239

/* use the PROC value to view the stack */

> ffffff21da3b68e0::findstack
stack pointer for thread ffffff21da3b68e0: ffffff00f88fc880
[ ffffff00f88fc880 _resume_from_idle+0xf1() ]
  ffffff00f88fc8b0 swtch+0x160()
  ffffff00f88fc960 turnstile_block+0x764()
  ffffff00f88fc9d0 rw_enter_sleep+0x1a3()
  ffffff00f88fca40 dsl_dataset_clone_swap+0x61()
  ffffff00f88fca90 dmu_recv_end+0x57()
  ffffff00f88fcc40 zfs_ioc_recv+0x31e()
  ffffff00f88fccc0 zfsdev_ioctl+0x10b()
  ffffff00f88fcd00 cdev_ioctl+0x45()
  ffffff00f88fcd40 spec_ioctl+0x83()
  ffffff00f88fcdc0 fop_ioctl+0x7b()
  ffffff00f88fcec0 ioctl+0x18e()
  ffffff00f88fcf10 sys_syscall32+0x101()

Here's a script to display all stacks for a given process type ...

For example: show all zfs stacks:

#!/bin/sh

for p in `pgrep zfs`; do
  echo "-------------------------"
  pargs $p
  echo "0t${p} ::pid2proc|::walk thread|::findstack" | mdb -k
done