cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« Previous day (Jan 19, 2006) | Main | Next day (Jan 20, 2006) »

Friday January 20, 2006

Little-known performance enhancements #3: the referential integrity plugin

The referential integrity plugin plays an important role in many Directory Server deployments. Whenever an entry is removed from the server, the referential integrity plugin can be used to ensure that any references to that entry (e.g., static groups in which the user was a member) are also removed. Similarly, if an entry is renamed with the modify DN operation, then those references are also renamed. Both of these things are desirable behaviors, so most customers have this plugin enabled in the server.

Every once in a while, we'll hear from customers that the performance of the server really lags when performing a lot of delete operations, and invariably the referential integrity plugin is to blame. While turning on this plugin does introduce a small amount of overhead, the real performance problems show up when it isn't tuned properly. Unfortunately, the default tuning is less than ideal, and most of the time a simple change can yield dramatic improvements.

First, a quick overview of how this plugin really works. It's actually quite simple: whenever an entry is deleted or renamed, the server performs an internal search to find any references to that entry and then removes or updates them accordingly. The internal search is based on a specific set of attributes, which is comprised of the values of the nsslapd-pluginargN attribute, where N is an integer greater than or equal to three. By default, this includes the member, uniqueMember, owner, seeAlso, and nsRoleDN attributes, although you can alter this list as necessary to suit your own needs (the attributes should all have a DN syntax).

In order to make the internal searches performed by the referential integrity plugin as fast as possible, the attributes that will be checked need to be indexed for equality. If you add your own attributes to the list then make sure that the corresponding equality indexes for those attributes are also in place. If not, then you'll find that every delete or rename will require a costly full DB scan while the plugin looks for references to the old DN. Conversely, if you're optimizing the indexes defined in the server then make sure that you don't remove any equality indexes for attributes contained in this list. Note that this applies to all backends in the server, so if you have more than one, then make sure they're all indexed appropriately.

If there is a mismatch between the set of attributes to check for the referential integrity plugin and the attributes that are indexed in the server, then you will definitely see problems. It may not be immediately obvious what the issue is, but every delete or modify DN operation will take a very long time to complete, and any time that happens the referential integrity plugin should be near the top of the list of possible causes. However, there is another configuration option that is much more easily overlooked because having it tuned sub-optimally won't cause it to rear its ugly head unless the server is under a barage of delete or modify DN operations: the update delay.

The referential integrity update delay is controlled by the nsslapd-pluginarg0 attribute. Its default value is zero, which means that all of the work of searching for and updating references to removed or renamed entries will be done by the same worker thread that was used to process the delete or modify DN operation. If you have multiple delete and/or rename operations in progress at the same time, then this means that multiple worker threads will be performing their search and modify operations concurrently as well. Due to contention between those threads, this can slow things down, and will also prevent those worker threads from being able to pick up new operations from the work queue. However, if you set this to a nonzero value, then the referential integrity plugin will create a background thread that will be used to perform this work. Whenever a delete or rename operation occurs, then instead of using the operation's worker thread to do the referential integrity work, information about that operation will be placed in a queue that will be processed the next time that background thread wakes up. The value of the update delay setting controls how long, in seconds, that thread will sleep between checks to see if there are any new delete or modify DN operations that have been processed.

Needless to say, the recommended value for this configuration attribute is something other than zero. It doesn't really matter much what that value is in terms of the overall performance of the server, so from that perspective a value of 1 will be about the same as 100. The only real difference is how long it will take for the updates to be processed so that old references will be cleaned up. This is something that you generally want to happen pretty quickly, so a value in the 1-10 second range will usually be preferred over larger values.

The nsslapd-pluginarg1 attribute controls the location of the referential integrity log file, which is used to hold information about recently-processed delete or modify DN operations that the referential integrity thread needs to look at the next time it wakes up. If you've moved your log files to another storage subystem (as I have previously recommended, at least if you're on a traditional filesystem like UFS or VxFS or EXT3), then you'll also want to update this path to use that location as well.

The nsslapd-pluginarg2 configuration attribute is no longer used and not something that you need to worry about when tuning the referential integrity plugin. In the older 4.x versions of the server, it was used to control whether the operations performed by the referential integrity plugin should be written to the changelog, but that was really only applicable for the single-master mechanism used by the 3.x and 4.x versions of the server. Today, the referential integrity plugin should be enabled on all masters, but you can leave it disabled on all read-only servers like hubs and consumers.

Posted by cn_equals_directory_manager ( Jan 20 2006, 01:00:18 PM CST ) Permalink

The Importance of RAID

It's been over a week since my last post, but I've been pretty busy since then, working with a customer to investigate a particularly large deployment. It was so large, in fact, that we weren't sure if we'd have the disk space to hold the database. Although we have a pretty impressive lab, especially given that it's primarily used for development, testing, and performance analysis, it's not quite as fancy as you can find in the iForce Customer Benchmarking Centers. It would have been great if we had a StorEdge 9990 System or something in that range, but we have much more modest arrays.

A common adage in storage goes something like "space, performance, and redundancy: pick any two". That is, in order to get two of them you have to sacrifice the third. Normally, we like to use RAID 1+0 (in which pairs of disks are mirrored, and then those mirrors are striped together). This is great for performance and redundancy, but it means that you only get the space of half the disks. In our largest arrays, RAID 1+0 only gives us a little over 800GB and we were concerned that might not be enough for our tests. As such, we had a decision to make -- either go with RAID 0 (create a stripe across all the disks, which is fast and big but offers no redundancy) or RAID 5 (use striping with parity to get the storage of all but one of the disks while still retaining the ability to survive a disk failure, although at a performance hit).

Our expert lab administrators wisely chose RAID 5, and while there was a significant performance penalty the redundancy paid off because we actually did lose a disk during the import but were able to continue without losing anything. The additional overhead of maintaining parity was a small price to pay compared with having to re-start the import.

In this case, we were constrained to currently-available production-quaility hardware and software. However, in the future ZFS will provide a great alternative. Because ZFS offers compression at the filesystem level, and because directory data compresses pretty well, we can get more space out of the same disks and therefore RAID 1+0 will be an option more of the time. And because compression often provides better performance because there's less I/O involved, it's a win-win situation.

Posted by cn_equals_directory_manager ( Jan 20 2006, 12:19:16 AM CST ) Permalink Comments [4]