cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« Previous month (Dec 2005) | Main | Next month (Feb 2006) »

Wednesday February 22, 2006

The Basics of Directory Server Cache Sizing

To the untrained eye, it might seem as if I hadn't written anything in a while. However, that's not entirely true. I'm always writing tons of code, but that isn't all that publicly visible. And while it is true that nothing new has shown up here for a couple of weeks, it doesn't mean that I haven't been writing anything. Hence this post.

I had recently done a week-long series on frequently-asked questions, and I had thought of doing that again. Unfortunately, the topic that I chose was one that I had been avoiding for a while because it's somewhat complex: deciding how best to size the server caches. So I started doing a write-up for it and quickly realized that it was going to take more than a week to cover everything that I wanted to include. I also didn't think that it was fair to release a small chunk at a time because it could just trigger questions that I had intended to answer in a later section. So I decided that it was best to write it all up into a document and post it all at once.

The document is available at http://blogs.sun.com/roller/resources/DirectoryManager/ds_cache_sizing.pdf.

Please note that there are all kinds of disclaimers to go with it. First of all, this is all based on testing and experience with the Directory Server 5.2 family of products. It does not necessarily reflect what's coming in future releases. In fact, there have been some recent changes in Directory Server 6 that will change things a little, and we've got other ideas for releases beyond that. Once we're closer to those releases, then I'll provide updates or amendments as necessary.

Another disclaimer is that the advice I'm giving is intended to be for the general case. There are usually special "corner cases" where something might be different or even counter-intuitive. I've tried to deal with that by explaining the underlying reasons for things so that you can better understand whether a given recommendation is applicable.

You'll notice that the document is completely devoid of formulas. That's intentional. Formulas are very often things that break down in special cases, and I'm a big fan of the "try it and see" approach. Hopefully there's enough here to let you draw your own conclusions to see what might work best for you. If there are things that I've missed, then I'll try to make sure it gets updated accordingly.

Posted by cn_equals_directory_manager ( Feb 22 2006, 09:58:39 AM CST ) Permalink

Wednesday February 08, 2006

Tuning the Number of Worker Threads

Since the first release of the T2000, and our Directory Server testing on it, I've received several questions about what kinds of special tuning are required. The answer is "not much". The only thing that we do differently on the T2000 than other systems is to increase the number of worker threads from the default of 30 to 32. On most other systems, we tend to reduce the number of worker threads from 30 to 24 (the nsslapd-threadnumber configuration attribute in cn=config). All the other tuning is the same for the T2000 as it is on other systems (SPARC and x86/x64).

So of course this raises the question "how do I figure out how to tune the number of worker threads?". I'm normally an advocate for the "try it and see" approach, and that's still pretty good advice in this case. However, this is a case in which you may not necessarily want to make performnce the highest priority.

A few years ago when we were first looking at Directory Server 5.2 performance on various platforms, we often found that the best number of worker threads was either equal to or a low multiple of the number of CPUs on the system. For example, for an 8-CPU V880 system, we found that 8 worker threads often gave slightly higher performance than any other configuration. I'm not talking about an earth-shattering difference, but it was at least measureable. However, you certainly don't want to set the number of worker threads to something as small as 8 because that can easily lead to a scenario in which you run out.

The most important thing to consider when configuring the number of worker threads in the Directory Server is the ratio between the rate at which the server receives requests from clients versus the rate at which it is able to process those requests. If requests come in faster than the server can process them, then you will find yourself in a scenario in which the server appears to be unresponsive. This isn't the case, but it has accumulated such a large backlog of work to do that whenever a new request comes in it may take a while for the server to be able to get to it.

For the fastest kinds of operations (e.g., indexed searches matching only one entry, binds, and compares), when the server is delivering the maximum performance possible, it's also often very close to 100% CPU utilization on the system, at least for most systems (you're not going to see it max out all 144-cores Sun Fire 25K). These are also the cases in which a direct relationship between the number of worker threads and the number of CPUs is the most visible. Under these conditions, adding more worker threads just increases the amount of contention and context switching, which will slow down the server somewhat. If you're 100% certain that these are the only kinds of operations you will ever perform, then it may be OK to consider a configuration like this, but given that these conditions never occur in the real world that's kind of a moot point.

For other operations that aren't as fast (e.g., substring searches, or anything that needs to perform a write), there's often some amount of CPU idle time because the system is waiting on disk I/O or network communication or some other non-CPU bottleneck. In these cases, adding more worker threads can be a good thing because under heavier load more requests can be performed in parallel since there's extra CPU to spare. This is the most helpful when the other operations to process won't further add to the underlying bottleneck that is causing the idle time to start with.

What you're most likely to find in a production environment is that most operations are very fast, and in in fact in many cases there are several worker threads idle at any given time. However, there will be an occasional inefficient operation like an unindexed search or something that takes a long time, and in those cases it's good to have the extra threads around so that processing on those doesn't interfere with the other, more efficient operations.

Every once in a while, we will have a customer complain that their server has hung. Upon closer investigation, 90% of the time, it's a case in which something is generating lots of inefficient operations that take several seconds or more to process, and there have been enough of them that all the worker threads have been consumed with them. Although you could attempt to deal with the problem by adding more worker threads, this is really just masking the problem and it usually doesn't work very well. The best thing to do is to identify the underlying problem and figure out how to configure the server to make it more efficient, or in some cases figure out how to change the client so it can make a more intelligent request. Unfortunately, at present this is a somewhat manual process and you may have to experience a hang for it to come to your attention. For future releases, we're working on enhancements to help identify these kinds of conditions quickly so that the problem can be dealt with in a more accurate and timely manner.

So overall, the best advice that I can offer is to look at the workload that the server is under. In many cases, the server can deliver more performance than customers actually need, so leaving the number of worker threads at the default may be perfectly acceptable. If you have a lot of inefficient operations that regularly take a second or more to process but the overall CPU utilization isn't usually completely pegging the system, then you'll probably want to look into having more threads available just in case a lot of these requests come in all at once (but please be reasonable -- you probably don't want hundreds of them because that can slow things down significantly and can also make debugging other problems a pretty unpleasant experience). If you absolutely need to squeeze every drop of performance out of the server that you can and you have a pretty efficient and tightly-controlled workload, then reducing the number of threads may help give you a little push.

Posted by cn_equals_directory_manager ( Feb 08 2006, 09:35:50 AM CST ) Permalink Comments [2]