Tuesday December 13, 2005
Tuesday December 13, 2005 Real-time Directory Server performance monitoring
As I've mentioned before, I spend a lot of my time measuring and analyzing the performance of the Directory Server. To help with that, I've developed tools like SLAMD that you can use to stress the Directory Server (or other applications) and the hardware it's running on to the limits and show you what kind of performance you can get out of it. That's really useful in a lot of ways because it can help us find hot spots in the code, and it can help customers understand the limits of what they can achieve.Of course, unless your production Directory Server instances are all under peak load 100% of the time, benchmarking won't give you a very good idea of the kind of demand the server is under at any given time. They can tell you the best that you might be able to achieve, but they can't tell you what you're getting right now. There are a few ways that you can get that kind of information, like analyzing the server access logs, but that's not really well-suited for real-time analysis.
To be honest, we don't really have anything inside the Directory Server (at least, not in current versions) that can provide you with information about the kind of load that the server is under in real time. But as it turns out, with Solaris 10 you don't need to wait for us to add something like that to the server because you can get it for yourself using the new Dynamic Tracing (DTrace) framework. You just need to know where to look. Or at least, you would need to know if I hadn't already done it for you. Using this simple DTrace script you can see what the Directory Server is doing in real time and with minimal impact on performance.
The dsstat.d script is a simple DTrace script that operates in the style of vmstat or iostat to show you what's going on inside the server. It prints a line of output once every second showing the total number of binds, searches, compares, adds, deletes, modifies, modify DNs, and unbinds that occurred over that one-second period (customizing it to use an interval other than 1 second is left as an exercise to the reader). It accomplishes this by using the PID provider to increment a counter every time the server enters a function used to process one of these kinds of operations, and then using the profile provider to print out a summary of this information once every second.
To use the script, simply use chmod to make it executable, and then run it using the PID of the Directory Server process as the only argument. For example, if the PID of your Directory Server process is "1234", then the command you would use is:
./dsstat.d 1234
The output of this tool looks like the following:
# ./dsstat.d 2906 TOTAL BIND SEARCH COMPARE ADD DELETE MODIFY MODDN UNBIND 92 0 69 0 4 1 18 0 0 454 0 369 0 11 10 65 0 0 441 0 352 0 22 6 60 0 0 485 0 405 0 18 10 52 0 0 391 0 317 0 18 6 50 0 0 529 0 433 0 17 9 68 0 0 528 0 432 0 17 15 60 0 0
I should point out that these numbers were obtained on my laptop under intentionally light levels of load for demonstration purposes only and are not representative of the actual performance you can achieve from the server. I should also point out that this script is provided "as-is" and without any warranty of any kind, so use it at your own risk. It should be entirely safe, but if for some reason it should have some undesirable side effect like horribly degrading performance or causing the server system to catch fire, then I'm not responsible.
I will say that under certain circumstances when I've been running this against large directory server processes (e.g., those consuming tens or hundreds of gigabytes of memory), DTrace can take a significant amount of time to start up, as I believe it has to walk the memory map associated with the Directory Server process and during this time the execution of the server is essentially suspended. So if you are running your server with big caches and are consuming lots of memory, you'll definitely want to see what kind of impact it has in a non-production environment before unleashing it on a live system. Since we've got more than a few customers that run their server under those conditions, it's something you'll want to watch out for.
Posted by cn_equals_directory_manager ( Dec 13 2005, 12:38:03 AM CST ) Permalink Comments [4]
Post a Comment:
Comments are closed for this entry.
| Archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Language | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Referrers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Today's Page Hits: 0 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Posted by William Hathaway on December 13, 2005 at 08:29 AM CST #
The truth is that it's probably good enough for most purposes if you have a script to grab all this information via LDAP and do the work. In the future, we're thinking about providing everything that you need to do this with no calculation required. But don't quote me on that.
Posted by Neil Wilson on December 13, 2005 at 11:39 AM CST #
Posted by Regu on December 13, 2005 at 09:23 PM CST #
The id2entry database file contains the entire contents of all the entries in that backend. All other database files are indexes used to help retrieve entries very quickly based on different sets of criteria. As far as the LDIF export goes, if you are generating a complete export, then there should not be any functional difference by only looking at the id2entry database. The main case in which other indexes may be helpful during the export process is when you are not performing a full export but rather are only interested in a particular subtree.
The db2ldif shell script (not the Perl variant) can be used both when the server is online and offline, and it does so using a separate process so it will minimize the impact on Directory Server performance during the export. If you use the db2ldif.pl, the export will be done from within the Directory Server process and it is generally more expensive. So I would recommend always using the db2lif shell script regardless of whether the server is online or offline.
If you are using the shell script variant and still see a significant impact on server performance, then about the only reason I can think of for that is that the disk I/O involved in the export is interfering with the disk I/O needed by the Directory Server to perform its processing. The best ways to avoid that are to ensure that the DB files generally reside in the filesystem cache so that there is no disk I/O required to read them, and to write the export to a separate disk subsystem so that writing the LDIF will not interfere with other server I/O. You should be able to use the iostat utility (e.g., "iostat -x -n 2") to watch the system during an export to see how much disk I/O there is. If you look in the %b column and see values up near 100, then that's bad because it means that particular disk is near 100% busy and you have an I/O problem.
If disk I/O isn't an issue, then I can't think of a very good reason that the LDIF export would have a significant impact on server performance. If nothing else, you could try to re-nice the export process so that it is given a lower priority than other things happening on the system. Or if you've got a lot of CPUs, then maybe you could create a one-CPU processor set and run the export in that.
Posted by Neil Wilson on December 13, 2005 at 10:57 PM CST #