cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« Previous day (Dec 12, 2005) | Main | Next day (Dec 13, 2005) »

Tuesday December 13, 2005

Real-time Directory Server performance monitoring

As I've mentioned before, I spend a lot of my time measuring and analyzing the performance of the Directory Server. To help with that, I've developed tools like SLAMD that you can use to stress the Directory Server (or other applications) and the hardware it's running on to the limits and show you what kind of performance you can get out of it. That's really useful in a lot of ways because it can help us find hot spots in the code, and it can help customers understand the limits of what they can achieve.

Of course, unless your production Directory Server instances are all under peak load 100% of the time, benchmarking won't give you a very good idea of the kind of demand the server is under at any given time. They can tell you the best that you might be able to achieve, but they can't tell you what you're getting right now. There are a few ways that you can get that kind of information, like analyzing the server access logs, but that's not really well-suited for real-time analysis.

To be honest, we don't really have anything inside the Directory Server (at least, not in current versions) that can provide you with information about the kind of load that the server is under in real time. But as it turns out, with Solaris 10 you don't need to wait for us to add something like that to the server because you can get it for yourself using the new Dynamic Tracing (DTrace) framework. You just need to know where to look. Or at least, you would need to know if I hadn't already done it for you. Using this simple DTrace script you can see what the Directory Server is doing in real time and with minimal impact on performance.

The dsstat.d script is a simple DTrace script that operates in the style of vmstat or iostat to show you what's going on inside the server. It prints a line of output once every second showing the total number of binds, searches, compares, adds, deletes, modifies, modify DNs, and unbinds that occurred over that one-second period (customizing it to use an interval other than 1 second is left as an exercise to the reader). It accomplishes this by using the PID provider to increment a counter every time the server enters a function used to process one of these kinds of operations, and then using the profile provider to print out a summary of this information once every second.

To use the script, simply use chmod to make it executable, and then run it using the PID of the Directory Server process as the only argument. For example, if the PID of your Directory Server process is "1234", then the command you would use is:

./dsstat.d 1234

The output of this tool looks like the following:

# ./dsstat.d 2906
TOTAL   BIND    SEARCH  COMPARE ADD     DELETE  MODIFY  MODDN   UNBIND
92      0       69      0       4       1       18      0       0
454     0       369     0       11      10      65      0       0
441     0       352     0       22      6       60      0       0
485     0       405     0       18      10      52      0       0
391     0       317     0       18      6       50      0       0
529     0       433     0       17      9       68      0       0
528     0       432     0       17      15      60      0       0

I should point out that these numbers were obtained on my laptop under intentionally light levels of load for demonstration purposes only and are not representative of the actual performance you can achieve from the server. I should also point out that this script is provided "as-is" and without any warranty of any kind, so use it at your own risk. It should be entirely safe, but if for some reason it should have some undesirable side effect like horribly degrading performance or causing the server system to catch fire, then I'm not responsible.

I will say that under certain circumstances when I've been running this against large directory server processes (e.g., those consuming tens or hundreds of gigabytes of memory), DTrace can take a significant amount of time to start up, as I believe it has to walk the memory map associated with the Directory Server process and during this time the execution of the server is essentially suspended. So if you are running your server with big caches and are consuming lots of memory, you'll definitely want to see what kind of impact it has in a non-production environment before unleashing it on a live system. Since we've got more than a few customers that run their server under those conditions, it's something you'll want to watch out for.

Posted by cn_equals_directory_manager ( Dec 13 2005, 12:38:03 AM CST ) Permalink Comments [4]