Chris Quenelle's Weblog
Thoughts on developer tools.

Friday May 12, 2006

Solaris Day at the University of Washington

I gave some presentations at a recent Solaris Day at the University of Washington in Seattle, so I figured I would give a very brief trip report. I talked about the Sun Studio tools in general (slides), and I also talked about developing multi-threaded apps on Solaris (slides). The slide sets were created by someone other than me, I can't take credit for very much work here, I basically just got up and talked for a while.

We had about 40 people signed up, but only about 10 people showed up. Some were system admins, some were software developers, and some didn't raise their hands for either of those, so they consider themselves something else. For example, one guy I spoke to was responsible for performance troubleshooting on production systems.

I was disappointed that the event wasn't advertised better. It wasn't mentioned anywhere on sun.com, and we had one student show up near the end who would have attended the whole thing if he had known it was being held. We had much more food and give-away items than we needed.

Most of the discussion (as I expected) was focussed on Solaris features, rather than software development. Because I know a lot about Solaris compared to the average Sun Studio engineer, I had a good time discussing Solaris topics with the attendees and with the other presenters. dtrace, ZFS, Xen, Containers(zones) were all asked about enthusiastically at least once.

I had a very good dinner with Angelo (dtrace guy) and Frank Curran (all-around Solaris-Days guy). We discussed ideas for the developer portal and how to bring dtrace into the reach of the masses among other topics.

Getting started with dtrace

Vijay forwarded me an email from Eugene and here's what I wrote:

Eugene wrote:
I have a code and I'd like to figure out where the user code is
when brk is being called.  Can this be done with truss?  If it's
done with dtrace I need some serious handholding.  Canned
scripts (or whatever) would be nice since I need to resolve this
in a hurry.

Vijay wrote:
Eugene, this is a perfect opportunity to refer you to ChrisQ's blog:
http://blogs.sun.com/roller/page/quenelle/200503
This one talks about doing exactly the thing you are asking about

Here is what I recommend:

1. download/untar the DTrace toolkit

2. run this command to see the stacks of all places that call brk

./DTraceToolkit-0.96/Bin/dtruss -s -t brk /bin/ls

Unless you give yourself dtrace permissions in /etc/user_attr, you will need to be root to run dtrace. The toolkit has a bunch of scripts in it that do wonderful things. And all without knowing anything about dtrace.

I went ahead and fiddled with dtrace for a bit, and here's what I came up with:

% dtrace -n 'pid$target::sbrk:entry { @num[ustack()] = count()}' -c "find /usr -name 'xyzzy'"

dtrace: description 'pid$target::sbrk:entry ' matched 1 probe
^C
dtrace: pid 17201 terminated by SIGINT

              libc.so.1`sbrk
              libc.so.1`_morecore+0x24
              libc.so.1`_malloc_unlocked+0x1fc
              libc.so.1`_smalloc+0x4c
              libc.so.1`malloc+0x4c
              libc.so.1`calloc+0x58
              libc.so.1`textdomain+0x38
              find`main+0x1c
              find`_start+0x108
                1

              libc.so.1`_morecore+0xdc
              libc.so.1`_malloc_unlocked+0x1fc
              libc.so.1`_smalloc+0x4c
              libc.so.1`malloc+0x4c
              libc.so.1`calloc+0x58
              libc.so.1`textdomain+0x38
              find`main+0x1c
              find`_start+0x108
                1

The number after each stack shows the number of times that stack trace was encountered.

Oh yeah. Did I tell you how much I hate the wysiwyg editor I'm using in Roller? Of course, once I get the content in the little box, it's too much trouble to change editors. I took me almost as long to get the preformatted text right as to write the email, figure out the script and write the rest of this blog.

Silly me. I forgot about using Xinha! It's in my firefox, but I forgot all about it.  I'm using it to add this last paragraph, and it's working fine.