cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« The Importance of... | Main | Making the Directory... »

Friday January 20, 2006

Little-known performance enhancements #3: the referential integrity plugin

The referential integrity plugin plays an important role in many Directory Server deployments. Whenever an entry is removed from the server, the referential integrity plugin can be used to ensure that any references to that entry (e.g., static groups in which the user was a member) are also removed. Similarly, if an entry is renamed with the modify DN operation, then those references are also renamed. Both of these things are desirable behaviors, so most customers have this plugin enabled in the server.

Every once in a while, we'll hear from customers that the performance of the server really lags when performing a lot of delete operations, and invariably the referential integrity plugin is to blame. While turning on this plugin does introduce a small amount of overhead, the real performance problems show up when it isn't tuned properly. Unfortunately, the default tuning is less than ideal, and most of the time a simple change can yield dramatic improvements.

First, a quick overview of how this plugin really works. It's actually quite simple: whenever an entry is deleted or renamed, the server performs an internal search to find any references to that entry and then removes or updates them accordingly. The internal search is based on a specific set of attributes, which is comprised of the values of the nsslapd-pluginargN attribute, where N is an integer greater than or equal to three. By default, this includes the member, uniqueMember, owner, seeAlso, and nsRoleDN attributes, although you can alter this list as necessary to suit your own needs (the attributes should all have a DN syntax).

In order to make the internal searches performed by the referential integrity plugin as fast as possible, the attributes that will be checked need to be indexed for equality. If you add your own attributes to the list then make sure that the corresponding equality indexes for those attributes are also in place. If not, then you'll find that every delete or rename will require a costly full DB scan while the plugin looks for references to the old DN. Conversely, if you're optimizing the indexes defined in the server then make sure that you don't remove any equality indexes for attributes contained in this list. Note that this applies to all backends in the server, so if you have more than one, then make sure they're all indexed appropriately.

If there is a mismatch between the set of attributes to check for the referential integrity plugin and the attributes that are indexed in the server, then you will definitely see problems. It may not be immediately obvious what the issue is, but every delete or modify DN operation will take a very long time to complete, and any time that happens the referential integrity plugin should be near the top of the list of possible causes. However, there is another configuration option that is much more easily overlooked because having it tuned sub-optimally won't cause it to rear its ugly head unless the server is under a barage of delete or modify DN operations: the update delay.

The referential integrity update delay is controlled by the nsslapd-pluginarg0 attribute. Its default value is zero, which means that all of the work of searching for and updating references to removed or renamed entries will be done by the same worker thread that was used to process the delete or modify DN operation. If you have multiple delete and/or rename operations in progress at the same time, then this means that multiple worker threads will be performing their search and modify operations concurrently as well. Due to contention between those threads, this can slow things down, and will also prevent those worker threads from being able to pick up new operations from the work queue. However, if you set this to a nonzero value, then the referential integrity plugin will create a background thread that will be used to perform this work. Whenever a delete or rename operation occurs, then instead of using the operation's worker thread to do the referential integrity work, information about that operation will be placed in a queue that will be processed the next time that background thread wakes up. The value of the update delay setting controls how long, in seconds, that thread will sleep between checks to see if there are any new delete or modify DN operations that have been processed.

Needless to say, the recommended value for this configuration attribute is something other than zero. It doesn't really matter much what that value is in terms of the overall performance of the server, so from that perspective a value of 1 will be about the same as 100. The only real difference is how long it will take for the updates to be processed so that old references will be cleaned up. This is something that you generally want to happen pretty quickly, so a value in the 1-10 second range will usually be preferred over larger values.

The nsslapd-pluginarg1 attribute controls the location of the referential integrity log file, which is used to hold information about recently-processed delete or modify DN operations that the referential integrity thread needs to look at the next time it wakes up. If you've moved your log files to another storage subystem (as I have previously recommended, at least if you're on a traditional filesystem like UFS or VxFS or EXT3), then you'll also want to update this path to use that location as well.

The nsslapd-pluginarg2 configuration attribute is no longer used and not something that you need to worry about when tuning the referential integrity plugin. In the older 4.x versions of the server, it was used to control whether the operations performed by the referential integrity plugin should be written to the changelog, but that was really only applicable for the single-master mechanism used by the 3.x and 4.x versions of the server. Today, the referential integrity plugin should be enabled on all masters, but you can leave it disabled on all read-only servers like hubs and consumers.

Posted by cn_equals_directory_manager ( Jan 20 2006, 01:00:18 PM CST ) Permalink

Comments: