Hacking Sun Java System Web Server pblocks using dtrace.

In my previous blog, I showed how to use NSAPI plugin and dtrace to write several monitoring tools. In this blog I will show that it is even possible to do few things with dtrace without having any NSAPI plugin installed into your web server. Yes that is true, dtrace can be used with stock web server without any configuration changes. Here is my watchpblocks.d dtrace script output (when I send a "/" request to my Web Server):

# dtrace -qs watchpblocks.d  11463
Req->vars : ntrans-base="/var/www" path="/var/www/" required-rights="list" content-length="1912"
Req->reqpb : clf-request="GET / HTTP/1.1" method="GET" protocol="HTTP/1.1" uri="/"
Req->headers : user-agent="curl/7.16.1 (sparc-sun-solaris2.8) libcurl/7.16.1 
                  OpenSSL/0.9.8d zlib/1.2.3 libidn/0.5.19" host="chilidev4.red.iplanet.com" 
                  accept="*/*" content-type="text/html" status="200 OK" 
                  transfer-encoding="chunked" content-length="2003"

11463 is the child process id of my test Web Server. Here is the output of wstop2.pl (similar to my previous version wstop.pl)

# perl wstop2.pl -d 5 11463
12:49:55  Requests: 3     (    0/sec) Bytes: 5736(  1147/sec)
Requests: GETs: 3      POSTs: 0      HEADs: 0      TRACE: 0
Responses: 1xx:0      2xx:3      3xx:0      4xx:0      5xx:0

Requests  Reqests/sec  Bytes Sent  URI
3         0            5736         /
^C

So how does it work. If you look at the dtrace script, you will find :

pid$1::flex_log:entry
{
...
}

So at the end of each request webserver calls flex_log method to log the request, at that time above dtrace probe is fired. As with most NSAPI functions Request and Session structure pointers are passed as an argument. dtrace script parses the structure and try to decode the pblocks. For this technique to work users need to have accesslog enabled which is almost always enabled.

The interesting aspect is that we don't need to do any configuration changes.

Caution : Since dtrace doesn't allow "for" loop or if/else logic so the pblock hash decoding is a complete hack. It may not work on all scenarios. Also on busy systems, lots of dtrace probes might be missed using this method. Since we are copying data from kernel land to user land several times.

Previous dtrace version, was much more lightweight as far as dtrace work is concerned. If future version of dtrace provides if/else and loop constuct inside dtrace script then the script could be improved and make it more reliable.

Also, right now these scripts will only work for 32 bit web servers. Here are the scripts :
watchpblocks.d
wstop2.pl
wbdtrace.pm

Posted at 02:18PM Aug 05, 2008 by Basant Kukreja in Sun Web Server  |  Comments[1]