cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« I've put it off long... | Main | Little-known perform... »

Wednesday December 07, 2005

Little-known performance enhancements #1: Transaction batching

This is the first of what I hope to make a series of posts about things you can do to help improve Directory Server performance, particularly with regard to settings that you may not be aware of. Some of these may touch on much more prominent configuration options where there is a lot of room for discussion (e.g., cache sizing), but I'll probably save those for when I have a good bit of time to cover them in appropriate detail.

The first topic I want to touch on is that of transaction batching. Historically, directory servers have been used much more heavily for read operations than for writes, with searches, binds, and compares vastly outnumbering adds, modifies, and deletes. In those cases, write performance isn't necessarily all that important. However, that is changing for a few reasons:

Directories are becoming more heavily used. As they need to handle higher and higher numbers of requests, even a relatively low percentage of write operatiosn can start to add up.
Directory data is becoming more dynamic. The percentage of write operations is increasing, and some popular directory-enabled apps include a write in most sequences of operations.
The amount of data synchronization is increasing. Products like Identity Synchronization for Windows, Identity Manager, and home-grown applications can merge data from multiple repositories, so a change in one is reflected in the other.
Although it happens less frequently, if large companies or organizations merge together, they've often got a flurry of activity combining their data into a single repository.

All of this adds up to the need for higher write performance rates than may have been required in the past. Even though the out-of-the-box tuning may be sufficient for many environments, when it isn't then there may be a few things that can be done to get some pretty significant boosts. One of the less well-known configuration options is the ability to use transaction batching, however it can have a very significant impact on performance. In our performance testing, enabling transaction batching can often double your write rate (of course, individual mileage may vary).

Transaction batching is basically a means of grouping multiple write operations together. In its default configuration, the Directory Server uses full durability, which means that by the time the response is sent to the client the server has verified that the data is committed to disk. It does this using the fsync(3C) call, which has historically been pretty heavyweight. Even though recent versions of Solaris have made improvements, it can still be an expensive operation, so reducing its frequency can dramatically improve performance.

One way to accomplish this is to disable transaction durability in the server by setting the nsslapd-db-durable-transactions attribute to "off". However, this is somewhat risky because if the worst should happen (e.g., an application or system crash) before the record of the write operation(s) get committed, there is basically no limit to the amount of change information that could be lost. The integrity of the database itself will remain intact, but recent changes (possibly any updates since the last checkpoint) may be lost. This is not a very desirable behavior, and even though it can help performance a lot it is generally not seen as a viable option for use by customers.

Transaction batching serves as a middle ground between full durability and no durability. In particular, it gives you control over the exposure that you might have in the event of a failure. To enable transaction batching, provide a nonzero value for the nsslapd-db-transaction-batch-val configuration attribute, where the value of that attribute controls the number of changes that may be grouped together under a single fsync. We typically use a transaction batch value of 5, which provides a significant performance improvement at a relatively low risk. In this case, you're at risk of losing information from at most four changes if an unexpected outage should occur. However, this risk can be further limited by other factors. The decision about when to commit changes to disk is based primarily on the following factors:

The configured transaction batch value. If you've configured a batch value of N, then the server will call fsync whenever the Nth oustanding change is performed, meaning that at most N-1 changes may be at risk.
An internal timer that will commit any outstanding changes if it has been 1000 milliseconds since the last fsync.
The transaction log buffer size. This buffer is used to hold the set of outstanding changes before they are committed. If this buffer gets full, then the changes will be immediately flushed to disk.

Most of the customers that I've talked to about transaction batching are willing to accept this relatively small risk of change loss because of the large performance benefits that it can provide. However, if this risk is not acceptable in your environment, then you may still be able to benefit from this capability by enabling transaction batching only when you need a little extra boost it and keeping it off the rest of the time to minimize exposure. To do that, change the transaction batch value from its default of zero to one. These values have the same semantic behavior (all changes will be immediately committed to disk before returning the result to the client), but the latter causes the server to use a different code path where it is possible to apply these changes on the fly without the need to restart, whereas a batch value of zero can only be changed by shutting down and restarting the Directory Server. Then, if you know that you have a lot of write operations that need to be performed and want the extra performance, you can use a tool like ldapmodify to make the configuration change over LDAP to temporarily enable batching, and then change it back to one when it is no longer needed.

Another quick note that I should make in this case is about the transaction log buffer size. As I mentioned above, the server will automatically commit any outstanding changes if the transction log buffer gets full, so you will generally want to make this big enough to hold all the outstanding changes to prevent flushing too frequently. The transaction log buffer size is controlled by the nsslapd-db-logbuf-size attribute, and its value is specified in bytes. Earlier versions of the server used a default of 32KB, but as of 5.2 patch 3 and later, it now uses 512KB. This should be OK in most cases, but if you have particularly large entries then you may want to increase it even further. You can make the log buffer size up to 25% of the transaction log file size (which is 10MB by default, so that means a maximum transaction log buffer size around 2.5MB).

Posted by cn_equals_directory_manager ( Dec 07 2005, 12:38:24 AM CST ) Permalink

Comments: