Jon Haslam's Weblog

Just before Christmas, I was unfortunate enough to do a slot at the London Opensolaris User Group. I say unfortunate because my session was supposed to be about DTrace Internals. Actually, the word Internals is far too grand a word for what I aimed to do. Really I should replace Internals with "a bit more under the hood than you'd normally go...". Anyway, I digress.

I was also unfortunate because Adam Leventhal was in London at the time and came along to lend a hand! My presentation itself (the material) was pretty bad really owing to the lack of time I'd had to put into, what is essentially, a very large topic. However, I hope I wasn't too dull and I hope that I wasn't too innacurate in my description of how little bits of DTrace work.

The vast topic of how DTrace actually works is hugely interesting and I'll do my best to put some of my pitted understanding up here in the not-too-distant future. One thing that I put as an addendum to my presentation was some mdb(1) incantations that allow you to extract a bit of information about what is going on around DTrace itelf.

DTrace has several mdb(1) dcmds of interest that allow us to shine some light on the workings of DTrace. Firstly, let's take a couple of the simple ones starting with the ::dtrace_state dcmd:

# mdb -k
Loading modules: [ unix krtld genunix specfs dtrace ufs ip sctp usba uhci s1394 fctl nca lofs audiosup zfs random nfs sppp ptm ipc crypto logindmux ]
> ::help dtrace_state

NAME
  dtrace_state - print active DTrace consumers

The ::dtrace_state dcmd actually has a very good description which says it all really. You can use it to display all or individual consumers that are running on the system. Used without any arguments we see:

> ::dtrace_state
            ADDR MINOR             PROC NAME                         FILE
ffffffff8155c800     2 ffffffff856be048 dtrace           ffffffff857a84d0
ffffffff8155c000     3 ffffffff856af190 dtrace           ffffffff857a8498

So, we have two consumers running. The intersting bit is the address given on the ADDR column. This is a pointer to a dtrace_state structure. The dtrace_state structure is of huge interst to us when figuring out what a consumer is doing as it holds kernel state for a consumer.

We can then use the ::dtrace dcmd to extract any data that is being held in kernel for that consumer (data that hasn't yet been read by the consumer). Let's take the second consumers dtrace_state address and feed in into the dtrace dcmd:

> ffffffff8155c000::dtrace
  fsflush                                                           1
  Xorg                                                              2
  battstat-applet-                                                  3
  gnome-netstatus-                                                  3
  nautilus                                                          3
  dtrace                                                            4
  mixer_applet2                                                     4
  soffice.bin                                                       4
  mdb                                                               5
  powernowd                                                         6
  java                                                             34
  sched                                                           100

It would be very interesting to know what probes were enabled and what the script looked like that generated that output. Well, that's where things get a bit tougher and I'll do that one in another entry.

I'll leave you with a small piece of mdb that shows you what providers are loaded on your system at the moment. Now, you could do this with dtrace(1M) and a simple bit of awk but that would be dull :-) . If I remember, next time I'll try to explain what it does...

> 1::id2probe | ::print -t dtrace_probe_t dtpr_provider | ::list dtrace_provider
_t dtpv_next | ::print -t dtrace_provider_t dtpv_name 
char *dtpv_name = 0xffffffff83959078 "dtrace"
char *dtpv_name = 0xffffffff8527ed00 "pid689"
char *dtpv_name = 0xffffffff83db07c0 "syscall"
char *dtpv_name = 0xffffffff83db0790 "sdt"
char *dtpv_name = 0xffffffff83db07d0 "mib"
char *dtpv_name = 0xffffffff83db07b0 "io"
char *dtpv_name = 0xffffffff83db0660 "proc"
char *dtpv_name = 0xffffffff83db0690 "sched"
char *dtpv_name = 0xffffffff83db0650 "fpuinfo"
char *dtpv_name = 0xffffffff83db08f8 "vminfo"
char *dtpv_name = 0xffffffff83db0920 "sysinfo"
char *dtpv_name = 0xffffffff83db0928 "vtrace"
char *dtpv_name = 0xffffffff8316cb78 "profile"
char *dtpv_name = 0xffffffff83da8a28 "lockstat"
char *dtpv_name = 0xffffffff83db0900 "fbt"
char *dtpv_name = 0xffffffff831a5658 "fasttrap"