cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

Main | Next month (Jan 2006) »

Friday January 27, 2006

Frequently-Asked Questions #4: Default Access Controls

Q: I ran [security product X] against the Directory Server and it raised concerns about anonymous access to entries like cn=schema, cn=monitor and the root DSE (aka null DN). Is this something I should be worried about? Can I disable this anonymous access?

A: I'll refrain from naming the security product at fault in this case, and I have no direct experience with some of the worst offenders, but it's pretty clueless in the area of Directory Server security. I also frequently hear Sun's security experts grumbling about stupid behaviors in other areas like looking at file permissions, so I would strongly suggest that you take anything that any of these tools report with a grain of salt.

First, let me say that yes, it is possible to configure access controls to prevent anonymous access to these entries if that is something that you decide really is necessary. In fact, they all already have a default set of ACIs that provide access to these entries out-of-the-box. For each of these entries, simply replacing "ldap:///anyone" with "ldap:///all" will accomplish this, since "all" refers to all authenticated users whereas "anyone" is all plus anonymous. However, before you do this, you should consider the possible implications that it may have.

There are definitely valid reasons that a client might need to have anonymous access to the root DSE. The whole point of the root DSE is to provide clients with information about the capabilities of the Directory Server so that they can understand how to interact with it. This includes information about the types of supported controls and extended operations that are available, as well as the supported SASL mechanisms that might impact the manner that the client should use to authenticate. Therefore, any client that expects to have access to this information anonymously could break if the default permissions are altered.

Just as some clients expect to have anonymous access to the root DSE so that they can understand what capabilities the server might have, some client might expect to have anonymous access to the schema so that they can understand how to interpret the information that they retrieve from the server. It is true that if a client is sophisticated enough to be able to interpret different syntaxes and matching rules that it will probably also have the ability to use an authenticated connection to retrieve it in most cases, but that may not be possible for cases in which the client is retrieving the root DSE to determine the mechanisms that it can use to authenticate.

There is less precedent for allowing anonymous access to the cn=monitor entry since the LDAP specifications do not make any requirement for making this kind of information available. However, it is ideal for periodic health checks and some customers use it in this manner. If the application or device making this health check expects to be able to retrieve the information in cn=monitor anonymously, then it could stop functioning if this access is removed.

Ultimately, it all comes down to whether you feel that making this information available anonymously really is a security risk and whether removing it will adversely impact any legitimate clients. If you feel that there legitimate reason for concern in providing clients with access to this information, then you should configure a test environment to remove these permissions and determine whether it will cause any problems with any of your clients. If there are problems, then you should evaluate the effort it will take to "fix" those clients versus the risk associated with making this information available. In some cases, you may be able to reestrict anonymous access to just the systems running those clients that need it (by either IP address or hostname).

Personally, I'm not aware of any problem that could arise from leaving these access controls in their default configurations. None of the information that the server makes available in this manner should make it significantly easier to attack the server, and clients that you may currently have (or may add in the future) might have a legitimate reason to require access to this information without authentication. Security by obscurity is rarely the correct approach, and if at any point a vulnerability is discovered that could be easier to exploit by the information provided by these entries, then the correct solution is to fix the underlying vulnerability rather than trying to hide it with access controls.

Posted by cn_equals_directory_manager ( Jan 27 2006, 07:53:07 AM CST ) Permalink

Thursday January 26, 2006

Frequently-Asked Questions #3: Log File Permissions

Q: Can I configure the permissions used to create the access/error/audit log files?

A: Not in current releases. This frequently-requested feature will be available in the upcoming 6.0 release, but it is not currently available in the 5.x server.

The reason that this feature is so frequently-requested is that the Directory Server is currently hard-coded to create log files owned by the user defined in the nsslapd-localuser configuration attribute with permissions of 0600. This means that only this user and the root account (or on Solaris 10, any account with the file_dac_read permission) will be able to read this file. Many customers would like to be able to set the permissions to 0640 so that anyone in the same group as the Directory Server user could at least read the file.

Until the 6.0 version arrives, there are a couple of things that can be done. Some customers have set up a cron job that periodically changes permissions on these files. While it is both simple and effective, it is kind of an ugly hack. Nevertheless, on some platforms it may be about the only effective solution. However, on Solaris a better approach is to run the Directory Server as a role rather than a normal user (I've actually already written about that here). In this case, any user that is a member of that role will be able to assume the role so that they can access the log files and perform any other administrative function that would otherwise require them to become the Directory Server user or would require root access.

See my previous post for all the details, but the basic means of converting the Directory Server user account to a role is to issue the command:

usermod -K type=role dirsrv

(replace "dirsrv" with the name of the account that you actually use for the Directory Server) and then assign a password for that role with the passwd command. Then, for each user that should be allowed to assume this role, issue the following command:

usermod -R dirsrv {username}

The ability to use roles in this manner has been around since Solaris 2.7, so it is available for all versions of Solaris on which we support the Directory Server.

Posted by cn_equals_directory_manager ( Jan 26 2006, 09:39:37 AM CST ) Permalink

Wednesday January 25, 2006

Frequently-Asked Questions #2: Disabling Non-SSL Access

Q: Normally, when I enable SSL in the Directory Server, I get both SSL and non-SSL listeners active. Is there any way to disable the non-SSL listener so that it will only accept SSL-based connections?

A: The short answer is "Yes", it is possible to do this. But there is a longer answer because there are a couple of ways to achieve the desired result.

If you really do want to completely disable non-SSL access to the server, then you can do so by setting the value of the nsslapd-port configuration attribute to zero. This isn't really publicly documented, but this behavior has existed for years and isn't likely to change in either the 5.x or the upcoming 6.x release. With this setting, then the server won't listen at all for non-SSL connections, so there is no way to accept connections from non-SSL clients.

Note, however, that if you do completely disable all SSL access, then certain things will stop working. This includes various administrative scripts that come with the Directory Server and expect to be able to communicate without SSL, including bak2db.pl, db2bak.pl, db2index.pl, db2ldif.pl, ldif2db.pl, ldif2ldap, monitor, ns-accountstatus.pl, ns-activate.pl, and ns-inactivate.pl. All of these scripts expect to be able to communicate with the server over the loopback address without using SSL. Similarly, other things like the administration server and console might need to be configured to use SSL for their communication with the server.

One way to deal with this requirement is to simply update all these components to ensure that they will be able to communicate over SSL before disabling non-SSL access. This isn't really that much work, since in most cases it simply requires finding the ldapsearch command line in the script and adding the appropriate options (in particular, adding -Z to enable SSL, adding -P to specify the path to the certificate database, and updating -p to point to the SSL-enabled port). However, you should be aware that these updates could possibly (although it is unlikely) be undone if you upgrade or patch the server and any changes have been made to these scripts over the previous version.

Another (and IMHO, better) approach that you can take is to leave non-SSL access enabled for the Directory Server but to constrain it so that it is only allowed for the loopback interface. You can achieve this by setting the value of the nsslapd-listenhost configuration attribute to "127.0.0.1". This will ensure that remote systems will not be able to communicate with the server without using SSL, but that local scripts will continue working as expected. Since no network communication will appear in the clear this still achieves the desired effect but in a much more convenient manner.

Posted by cn_equals_directory_manager ( Jan 25 2006, 09:07:59 AM CST ) Permalink

Tuesday January 24, 2006

Frequently-Asked Questions #1: Size Limits

This is the first entry in a series of answers to what I consider frequently-asked questions about the Directory Server. These are questions that I see repeatedly from customers on internal mailing lists, or hear directly from customers. They may run the gamut from simple to complex, and some of them may be RTFM-type questions. Nevertheless, I figured it would be a good idea to answer them here so that when they're asked again we can point them to this information (i.e., RTFB) rather than having to craft the same response every time.

Q: Are there any size limits in the Directory Server? What's the largest attribute name you can use? What's the largest attribute value? What's the largest entry? What's the largest number of values you can have for an attribute? How many entries can you have?

A: In general, there is no hard-coded limit for any of these. Some directory servers do have hard-coded limits, so it is a legitimate question. But for all practical purposes, there isn't much of a realistic limit that you will encounter. But there are some operational limits that may need to be tuned in a few cases.

The first such limit is the maximum size of a request that the server will handle. This is controlled by the nsslapd-maxBERSize attribute in the cn=config entry (by default, this isn't present, but a value of about 2MB will be enforced). The Directory Server will immediately reject any request that it receives which will be larger than this amount and terminate the connection to the client. This limit is enforced as a means of helping to prevent denial-of-service types of attacks in which the client tries to consume bandwidth and memory that could otherwise be used for handling other requests. The default 2MB limit is fine in most normal directory uses, but there are occasionally times when it might be a little too small. One common case would be a static group with a very large number of members (e.g., at least in the hundreds of thousands). Another case is that in which an entry might need to store large blobs of data (like statistics collected by SLAMD). In these cases, increasing the value of the nsslapd-maxBERSize configuration attribute (which does require a restart to take effect in current versions) will allow you to store larger amounts of information.

If you do increase the maximum request size, then the next limit that you may encounter will be that of the transaction log file size, which is controlled by the nsslapd-db-logfile-size attribute and is 10MB by default. You may not have a single entry that is greater than the transaction log file size, so if you expect to have entries that are larger than 10MB, then you may need to increase this setting. Note, however, that if you already have one or more transaction log files with the old size (which will always be the case if you've got an existing database, and you should *NEVER* delete these files), then the database will continue to use this size. Therefore, if it is necessary to increase this limit, then you will need to export all database contents to LDIF (in all backends), change the configured size, remove all existing database contents, and re-import from LDIF.

If you get beyond the transaction log file size, then the next limit that you should encounter would be the largest single ASN.1 element that the server can handle, which is constrained by the size of a 32-bit signed integer (2GB minus one byte). This limit isn't going away any time soon, but in the unlikely case that you do have entries that need to be larger than 2GB in size, then you'll have to split them up in some way.

All of the above are related to the maximum size of an entry (and therefore, the largest size of any component of an entry), but have no impact on the maximum number of entries that can be held in the server. At present, the maximum number of entries in any one backend is limited by the 32-bit entry ID (less a few special values that have particular meanings), so you will not be able to have more than 4,294,967,292 entries in any one database. At present, we don't have any customers even close to that limit, but if it did become a potential issue, then it would be posible to split the entries into multiple backends to circumvent this constraint. In the future, we will likely be using 64-bit values for the entry ID so that this will not be an issue.

Posted by cn_equals_directory_manager ( Jan 24 2006, 09:39:16 AM CST ) Permalink Comments [2]

Monday January 23, 2006

Making the Directory Server Dump Core

In general, the Directory Server has shown to be quite reliable. We've had customers report the server running in a production environment for over a year between crashes or restarts. While it's generally a good idea to apply upgrades and patches as they come out, hopefully there's no other reason to have to interrupt its operation. But from time to time a customer will run into a bug that could cause the server to crash. Nobody wants this to happen, but even worse than having the server crash due to a previously-undiscovered bug would be for it to crash multiple times before the cause can be identified and resolved.

On UNIX systems, if a process crashes then it can leave behind a core file. A core file is essentially a snapshot of all the memory associated with the process at the time of its demise. Using tools like pstack, dbx, or mdb, engineers can look into the core file and use the information it provides to determine the cause of the crash. Of course, this will only be helpful if a core file is actually generated, and there are a few reasons that this may not happen. It's in everyone's best interests to ensure that those reasons are addressed and resolved so that if a crash does occur it will provide enough information to give us a chance to figure out what went wrong.

The first thing that might prevent the server from dumping core would be if the process can't actually write the file. This can happen if either the process doesn't have write permission to the target directory, or if there isn't enough disk space available. The first of these used to be a problem with early versions of the 4.x server (it would try to place the core file in the filesystem's root directory), but all recent versions will attempt to write it into the same directory as the error log file. To address the second concern, you should make sure that there is enough disk space available to hold the core file. The core file size will be basically equal to the resident size of the Directory Server process, which you can find with the command:

ps -o rss -p {PID}

On Solaris systems, if you don't have enough space there and don't want to move your error log to another location, or if you just want the core file written somewhere else, then you can change the location using the coreadm command, although that command will be discussed in more detail later in this post.

Another reason that a core file might not be written would be if a resource limit has been set that could constrain the core file size. This is particularly common on many Linux systems, which for some reason seem to consider core files undesirable. I suppose that this might be the case if you're just resolved that something will crash from time to time and you'll just restart without any investigation, but it's a bad thing if you actually care about figuring out why the crash occurred. To determine whether your system will prevent core files from being generated, issue the command

ulimit -c

If the output of this command is anything other than "unlimited", then that value will be the maximum allowed size of a core file in 512-byte blocks. You can fix that by updating the start-slapd script so that the following apears near the beginning:

# Allow core files of any size
ulimit -c unlimited

Another very common reason that the Directory Server may not be allowed to dump core if it crashes is that it often runs as a setuid process. If you want the server to listen on a privileged port (e.g., 389 and/or 636) and you don't have the ability to use process rights management, then you will need to start it as the root user. For security reasons, if you do this then the server will later call setuid to drop root privileges and run as a different user. However, if this happens then that application will not be allowed to dump core because the core file would be written as the unprivileged user but may still have sensitive information from memory that was obtained while it was running as root (this isn't true for the Directory Server but may be true for other applications that operate in this manner).

Solaris provides a way of circumventing this problem through the use of the coreadm(1M) command. This command provides all kinds of useful capabilities for dealing with the way that core files are handled, and one of those options can be used to specifically allow setuid/seteuid/segid processes to dump core. For security reasons, the core files will only be readable by the root user, but this is a minor inconvenience when compared with not being able to get a core file at all. To enable set*id processes to dump core, issue the following commands:

coreadm -e proc-setid
coreadm -u

This is a global setting and only needs to be done once per system (the "coreadm -u" command updates a configuration file so that the setting will be applied automatically when the system boots).

As I mentioned above, the coreadm command has all kinds of features for tweaking core file generation, and I won't get into all of them here, but you can read the man page for all the details. However, I will address one more important option that you may want to consider using. By default, if the Directory Server crashes and dumps core, then it will create a file named "core" in the same directory as the error log file. If there's already a previous core file there, then it will be overwritten and any useful information it may have contained will be lost (this assumes that the existing file had permissions such that it could be overwritten; if not, then the old core file would be kept and the details of the new crash would be lost). This is generally considered a bad thing, so you probably want to avoid it. Further, depending on how much disk space you have available on that volume, you may not want to allow writing a core file in the logs directory but would rather have it appear somewhere else. Both of these can be accomplished on a per-process level using the "-p" option to coreadm. For example, when I'm setting up the Directory Server on Solaris, I always add the following near the beginning of the start-slapd script:

# Set the pattern to use for Directory Server core files.
coreadm -p core.%f.%p $$

This will cause core files generated by the Directory Server (actually, the "$$" is the PID of the current process which is the start-slapd script itself, but it will be inherited by child processes) to contain both the executable name and the process ID. So if the Directory Server was running with PID 1234, then the name of the core file generated if it crashes would be "core.ns-slapd.1234". The value to the -p argument can also contain path information, so if you wanted core files to appear in the "/export/cores" directory instead of the server logs directory, then you would use a pattern like "/export/cores/core.%f.%p" instead. Just make sure that the target directory has enough space to hold the core file and that permissions are configured correctly to allow the server to dump core.

Once you've got your system configured properly, you can test it by trying to make the Directory Server crash and dump core. Hopefully you won't come across any cases in which the server does this accidentally, but you can intentionally cause it to happen by issuing the command:

kill -11 {PID}

This will send a signal to the Directory Server that would be identical to what it would have received from the OS if it had been guilty of a segmentation fault. If everything is set up properly, then the server should immediately stop and dump core. Obviously, this is not something you want to try on a production system that is actively handling requests, but it is a good practice to follow when you're first setting up the server so that if you do encounter a crash at some point in the future that it will at least leave behind a core file that can be used to help figure out what went wrong so it doesn't need to happen again.

Posted by cn_equals_directory_manager ( Jan 23 2006, 07:59:12 AM CST ) Permalink Comments [2]

Friday January 20, 2006

Little-known performance enhancements #3: the referential integrity plugin

The referential integrity plugin plays an important role in many Directory Server deployments. Whenever an entry is removed from the server, the referential integrity plugin can be used to ensure that any references to that entry (e.g., static groups in which the user was a member) are also removed. Similarly, if an entry is renamed with the modify DN operation, then those references are also renamed. Both of these things are desirable behaviors, so most customers have this plugin enabled in the server.

Every once in a while, we'll hear from customers that the performance of the server really lags when performing a lot of delete operations, and invariably the referential integrity plugin is to blame. While turning on this plugin does introduce a small amount of overhead, the real performance problems show up when it isn't tuned properly. Unfortunately, the default tuning is less than ideal, and most of the time a simple change can yield dramatic improvements.

First, a quick overview of how this plugin really works. It's actually quite simple: whenever an entry is deleted or renamed, the server performs an internal search to find any references to that entry and then removes or updates them accordingly. The internal search is based on a specific set of attributes, which is comprised of the values of the nsslapd-pluginargN attribute, where N is an integer greater than or equal to three. By default, this includes the member, uniqueMember, owner, seeAlso, and nsRoleDN attributes, although you can alter this list as necessary to suit your own needs (the attributes should all have a DN syntax).

In order to make the internal searches performed by the referential integrity plugin as fast as possible, the attributes that will be checked need to be indexed for equality. If you add your own attributes to the list then make sure that the corresponding equality indexes for those attributes are also in place. If not, then you'll find that every delete or rename will require a costly full DB scan while the plugin looks for references to the old DN. Conversely, if you're optimizing the indexes defined in the server then make sure that you don't remove any equality indexes for attributes contained in this list. Note that this applies to all backends in the server, so if you have more than one, then make sure they're all indexed appropriately.

If there is a mismatch between the set of attributes to check for the referential integrity plugin and the attributes that are indexed in the server, then you will definitely see problems. It may not be immediately obvious what the issue is, but every delete or modify DN operation will take a very long time to complete, and any time that happens the referential integrity plugin should be near the top of the list of possible causes. However, there is another configuration option that is much more easily overlooked because having it tuned sub-optimally won't cause it to rear its ugly head unless the server is under a barage of delete or modify DN operations: the update delay.

The referential integrity update delay is controlled by the nsslapd-pluginarg0 attribute. Its default value is zero, which means that all of the work of searching for and updating references to removed or renamed entries will be done by the same worker thread that was used to process the delete or modify DN operation. If you have multiple delete and/or rename operations in progress at the same time, then this means that multiple worker threads will be performing their search and modify operations concurrently as well. Due to contention between those threads, this can slow things down, and will also prevent those worker threads from being able to pick up new operations from the work queue. However, if you set this to a nonzero value, then the referential integrity plugin will create a background thread that will be used to perform this work. Whenever a delete or rename operation occurs, then instead of using the operation's worker thread to do the referential integrity work, information about that operation will be placed in a queue that will be processed the next time that background thread wakes up. The value of the update delay setting controls how long, in seconds, that thread will sleep between checks to see if there are any new delete or modify DN operations that have been processed.

Needless to say, the recommended value for this configuration attribute is something other than zero. It doesn't really matter much what that value is in terms of the overall performance of the server, so from that perspective a value of 1 will be about the same as 100. The only real difference is how long it will take for the updates to be processed so that old references will be cleaned up. This is something that you generally want to happen pretty quickly, so a value in the 1-10 second range will usually be preferred over larger values.

The nsslapd-pluginarg1 attribute controls the location of the referential integrity log file, which is used to hold information about recently-processed delete or modify DN operations that the referential integrity thread needs to look at the next time it wakes up. If you've moved your log files to another storage subystem (as I have previously recommended, at least if you're on a traditional filesystem like UFS or VxFS or EXT3), then you'll also want to update this path to use that location as well.

The nsslapd-pluginarg2 configuration attribute is no longer used and not something that you need to worry about when tuning the referential integrity plugin. In the older 4.x versions of the server, it was used to control whether the operations performed by the referential integrity plugin should be written to the changelog, but that was really only applicable for the single-master mechanism used by the 3.x and 4.x versions of the server. Today, the referential integrity plugin should be enabled on all masters, but you can leave it disabled on all read-only servers like hubs and consumers.

Posted by cn_equals_directory_manager ( Jan 20 2006, 01:00:18 PM CST ) Permalink

The Importance of RAID

It's been over a week since my last post, but I've been pretty busy since then, working with a customer to investigate a particularly large deployment. It was so large, in fact, that we weren't sure if we'd have the disk space to hold the database. Although we have a pretty impressive lab, especially given that it's primarily used for development, testing, and performance analysis, it's not quite as fancy as you can find in the iForce Customer Benchmarking Centers. It would have been great if we had a StorEdge 9990 System or something in that range, but we have much more modest arrays.

A common adage in storage goes something like "space, performance, and redundancy: pick any two". That is, in order to get two of them you have to sacrifice the third. Normally, we like to use RAID 1+0 (in which pairs of disks are mirrored, and then those mirrors are striped together). This is great for performance and redundancy, but it means that you only get the space of half the disks. In our largest arrays, RAID 1+0 only gives us a little over 800GB and we were concerned that might not be enough for our tests. As such, we had a decision to make -- either go with RAID 0 (create a stripe across all the disks, which is fast and big but offers no redundancy) or RAID 5 (use striping with parity to get the storage of all but one of the disks while still retaining the ability to survive a disk failure, although at a performance hit).

Our expert lab administrators wisely chose RAID 5, and while there was a significant performance penalty the redundancy paid off because we actually did lose a disk during the import but were able to continue without losing anything. The additional overhead of maintaining parity was a small price to pay compared with having to re-start the import.

In this case, we were constrained to currently-available production-quaility hardware and software. However, in the future ZFS will provide a great alternative. Because ZFS offers compression at the filesystem level, and because directory data compresses pretty well, we can get more space out of the same disks and therefore RAID 1+0 will be an option more of the time. And because compression often provides better performance because there's less I/O involved, it's a win-win situation.

Posted by cn_equals_directory_manager ( Jan 20 2006, 12:19:16 AM CST ) Permalink Comments [4]

Tuesday January 10, 2006

Sun T2000 vs Dell 6850 Revisited

Last month, I wrote about a demo that we presented in Austin comparing LDAP authentication performance on the Sun Fire™ T2000 server (one UltraSPARC® T1 processor at 1.0 GHz and 32GB DDR2 memory) with that of the Dell PowerEdge™ 6850 server (four dual-core Intel® Xeon® EM64T processors at 3.2GHz and 32GB DDR2 memory), which is about the best that Dell has to offer. You can read that post for the details, but in short the Sun server (which is cheaper, smaller, and consumes a lot less power than the Dell system) won the race pretty handily.

In my not-entirely-modest opinion, this demo was pretty impressive, and that was the goal since it was a marketing-driven presentation. I also believe that it was entirely fair, because we did everything we could to get the best performance possible from each system. However, it was more a flashy "bells and whistles" kind of demo using a custom GUI client that I hacked together over a couple of days, so you may be inclined to take those results with a grain of salt.

After the demo was complete, we took the two systems back into our lab for some more formal performance analysis using SLAMD. We ran a whole gamut of tests on the systems with varying numbers of user entries. On the Dell system, we first ran the tests using 250 thousand user entries under Microsoft® Windows® Server 2003 x64 Edition, which was the OS that was pre-installed on that system and that it was running during the demo. Then, we installed Solaris on it and ran through the same set of tests with 250 thousand, 1 million, and 10 million users. We wanted to run through all these tests on Linux as well, but unfortunately we hit some time constraints because we were actually borrowing the Dell box from another group within Sun and they wanted it back for their own work. On the T2000 system, we ran the same set of tests on Solaris with 250 thousand, 1 million, and 10 million users.

During all this testing, we collected a lot of data. Although I can't really share much of it, I can offer the results of the LDAP authentication performance tests under these conditions. So here they are:

System Type	Operating System	Number of User Entries	Average LDAP Authentications per Second
Dell PowerEdge™ 6850	Windows	250,000	2984.908
Dell PowerEdge™ 6850	Solaris	250,000	4374.923
Dell PowerEdge™ 6850	Solaris	1,000,000	3744.291
Dell PowerEdge™ 6850	Solaris	10,000,000	2800.135
SunFire™ T2000	Solaris	250,000	6832.893
SunFire™ T2000	Solaris	1,000,000	6233.475
SunFire™ T2000	Solaris	10,000,000	4457.075

As you can see from this data, running Solaris on the Dell system yields much better performance than running Windows. In fact, it took a 40-fold increase in the number of user entries to cause the Directory Server on Solaris to drop down to what we had measured on Windows. You can also see that the T2000 was still the clear winner, even if you just look at the performance numbers and ignore how much money you can save while achieving these rates.

A few notes about this testing:

When we installed Solaris on the Dell system, we chose to use Nevada build 28 (which is available through Solaris Express) rather than Solaris 10. The primary reason for this was that we wanted to use ZFS to provide RAID capabilities for the internal disks. For the Windows testing, an on-board Adaptec hardware RAID controller was used for this, but when we installed Solaris 10 on this system we saw the individual disks rather than the hardware RAID. As this was a borrowed system and time to investigate this issue was limited, we chose the software RAID approach using ZFS. Other testing has not shown a significant performance difference for Directory Server between Solaris 10 and Nevada builds, and disk I/O was kept to a minimum with caching during these authentication performance tests so the underlying storage is largely irrelevant.
These tests were run with the SLAMD 1.8.2 LDAP Weighted AuthRate job, using an 80/20 weighting (80% of the authentications were targeted at a subset of 20% of the users, and the remaining 20% of the authentications targeted the remaining 80% of the users). This is the same workload used in the graphical demo. This was run as an optimizing job in order to find the best performance with each configuration, and the numbers reported here were taken from a re-run of the best iteration from the optimizing job.
Each authentication consisted of an anonymous search to find a user account based on a login ID, followed by a bind as that user to verify the password. In the graphical demo that we provided, searches to find the user account were performed as the root DN rather than anonymously in order to provide better performance by avoiding access control evaluation. This does not impact the fairness of the comparison in either case, since all servers involved in the comparisons were subject to the same workload.
For each of these tests, client load was generated using four V60x systems, each of which had two 2.8 GHz Xeon processors, running Solaris 10. In the graphical demo, we used a single V20z as the client for each system. In both cases, we were able to ensure that the client was not a performance bottleneck for the testing.
In the graphical demo, the entries loaded into each Directory Server instance were created using the example.template file provided with the MakeLDIF LDIF generator (which comes in the tools/MakeLDIF directory in the SLAMD server installation). Entries for the formal test were also generated using MakeLDIF, but with a different template that uses slightly larger entries and includes additional attributes and objectclasses that would normally be found in the user's entry in a Messaging Server installation.
The Directory Server instances in the graphical demo were each configured with access logging disabled for a slight performance increase, and also to avoid disk write operations because both systems were using internal storage during that demo. In the formal performance tests, access logging was enabled and written to RAID-based storage. Due to caching, no disk reads were required in any of the tests.
On the T2000 system, the 64-bit version of the Directory Server was used. On the Dell system, both the Solaris and Windows testing made use of the 32-bit version, since no 64-bit version is currently available for x64 systems. On Windows, this 32-bit means that the Directory Server was only able to directly access a maximum of 2GB of memory; on Solaris, the maximum was 4GB of memory. These constraints were sufficient for fully-caching the data for the tests with 250,000 users, but were not sufficient for the larger data sets. On Windows, we were unable to achieve acceptable results for larger data sets because of the caching constraints. On Solaris, the filesystem cache was used to help avoid disk reads when testing with the larger data sets.

Posted by cn_equals_directory_manager ( Jan 10 2006, 09:14:44 AM CST ) Permalink

Sunday January 08, 2006

What's happening with SLAMD?

If you are at all interested in performance testing Directory Server or other network applications, or if you have read much of my earlier posts, then you're aware of The SLAMD Distributed Load Generation Engine. You may also note that the last release of any kind was in June and that the SLAMD site doesn't change all that much. If you are subscribed to the SLAMD-Commit@SLAMD.com mailing list, then perhaps you've seen some of the stuff that's been going on since then. However, since it's sometimes hard to see the forest for the trees, and it's even harder to see stuff that hasn't been committed yet, then I thought I would give a brief overview of what's coming in the next release, which should be version 2.0.0.

The biggest change is the switch from a configuration directory to an embedded database, using the Berkeley DB Java Edition. Right now, SLAMD 1.x requires an LDAPv3 directory server, like the Sun Java System Directory Server or OpenLDAP, in order to store its configuration and all job information. For people that use SLAMD in order to test a Directory Server, or for testing something that requires a Directory Server like Access Manager or Portal Server, then this is no big deal. But if you're using it to test something else and you're not in the habit of using directory servers, then it can be more complex than some people want. By using an embedded database, the complexity can be almost completely hidden from the user. It also allows for better extensibility for the future, since using a Directory Server requires updates to the schema in order to support new attributes for new features, whereas this is not necessary for an embedded database.

The code for using the embedded database has actually already been implemented, and if you check out SLAMD from the CVS head, then you will be using that code. Note that the new database format is not backward compatible with the existing format, although there is a method to migrate data from a SLAMD 1.x configuration directory into the new SLAMD 2.x database. However, this is a one-way conversion so any jobs run with the SLAMD 2.x version of the code won't be accessible to SLAMD 1.x servers. Moving forward into the future, the design used for the database should be flexible enough to avoid the need for another big change like this.

The other major change between SLAMD 1.x and 2.x will be the protocol used to communicate between the SLAMD server and the various types of clients. Although the SLAMD 1.x protocol works quite well, it was not designed to be as extensible as necessary to add new features. Therefore, with a couple of exceptions that took quite a bit of work to implement, everything that a SLAMD 1.8.2 client can do could be done by a SLAMD 1.0 client (and in fact, quite a bit earlier than that). I've been able to come a long way just by implementing what have primarily been server-side enhancements, but it's time to be able to have new features on the client side as well, and to make it possible to have a much more extensible protocol so that another major redesign won't be necessary.

The code for the new protocol has mostly been written and checked into the repository. However, it's not actually in use because none of the client code has been updated to make use of it rather than the SLAMD 1.x protocol. I've spent a little time working on a new client to use the new protocol, but I've been a little distracted of late and therefore haven't been able to devote a whole lot of time to it. But once that work is done, then I'll probably release an alpha version of SLAMD 2.0 shortly thereafter.

All of the other changes that have been made so far are much less significant. The most interesting ones include:

I've made several minor updates to the UI to add niceties in a few places. For example, when a job is running, if the SLAMD server can figure out how much longer it thinks that a job will remain active, then it will display that, along with information about the clients and monitor clients that are being used to run it. Also, the status page now lists the five most recently completed jobs so you can quickly find them, which is handy if you have a server with lots of data in it.
I've added a "minimum percent improvement" feature to the provided optimization algorithms that can be used to indicate that a new iteration must be at least x% better than the previous best in order to be considered the new best. This can help avoid cases where optimizing jobs take a horribly long time to complete because new iterations make tiny, statistically-insignificant improvements over previous ones that cause new iterations to be scheduled.
I've made several minor changes and improvements in the area of resource monitors. I've improved the default configurations for several of the monitors so that they will be more likely what you want to see by default (or at least what I want to see). Also, I've added a new MPStat monitor that can measure CPU utilization on a per-CPU basis (Solaris only for now), and I've updated the IOStat monitor so that it can also include the percent busy information for the associated disks (also a Solaris-only improvement). I fixed a bug in the NetStat monitor that could cause it to come up with negative values whenever a counter rolled over.
I've added the ability to group resource monitor statistics by monitor class. This means that when graphing statistics for a job, you can see not only a "Graph all resource monitor statistics" checkbox, but also checkboxes like "Graph all IOStat statistics" and "Graph all VMStat statistics". Unfortunately, this depends on a feature in the new protocol and therefore its true potential has not yet been realized.
The HTTP client has been updated so that you can now configure it to automatically delete any cookie whose value is set to "LOGOUT". Although I can't find this in any official HTTP specification, apparently most browsers exhibit this behavior and some Web-based applications depend on it. This allows the SLAMD HTTP client to be more compatible with those applications. Also in the area of cookies, support has been added for the cookie comment and version fields, and for deleting cookies by name. Not related to cookies in any way, a socket timeout option has been added that makes it possible to require a response in a specified length of time, and also the local port to use for outbound sockets can be specified.
I've added better support for read-only mode such that information that the SLAMD server considers "sensitive" won't be displayed. This includes things like addresses of clients and resource monitor clients used to run a job and the e-mail addresses of users to notify when the job is complete. I've also updated the various types of parameters so that they can be declared sensitive and therefore the information they contain will not be displayed in this mode (password parameters are marked sensitive by default).

It's likely that most of the other improvements in SLAMD 2.0.0 will be pretty minor. I've still got some big ideas that will be coming in the future, but the flexibility of the new protocol will allow me to push them out to SLAMD 2.0.1 or SLAMD 2.1 without having to worry about backward compatibility as much as was necessary in the past.

As for the big question as to when SLAMD 2.x will be available, that's still to be determined, but the generic answer is "as soon as I'm comfortable with what's in it and it's gone through enough testing that I feel it's stable enough". Some of that uncertainty is simply because SLAMD is something that I work on in my spare time, and I don't usually have a whole lot of that. I will admit that I didn't get as much work done on it as I would have liked during the recent Christmas break, but writing the code for the new protocol took me quite a bit less time than I expected so that's at least a little bit of a trade-off. And of course if you want to see what's there now, it should be possible to build it for yourself from what's checked in, and if you find any bugs, then I'll do my best to fix them as quickly as possible. It may not have all the bells and whistles that I hope it will have in the not-too-distant future, but it should be functional and hopefully a little better than what's available in SLAMD 1.8.2. However, I'd still caution you that if you've got some big testing to do, then it might be safer to stick with what's officially released.

Posted by cn_equals_directory_manager ( Jan 08 2006, 11:57:53 PM CST ) Permalink Comments [2]

Tuesday January 03, 2006

Why you can't accelerate LDAP over SSL

As information security comes more and more to the forefront, we're seeing an ever-increasing percentage of our customers configure most or all of their directory-enabled applications to communicate with the server over SSL. It's common knowledge that protocols tunnelled through SSL are generally more expensive than if they're used without SSL, so many of these customers ask about using an SSL accelerator to help reduce this impact. However, all the administrators that I've talked to that have used an accelerator in conjunction with the Directory Server have been unimpressed with the results. For those that I've talked to ahead of time, I've been able to explain why this is the case and let them decide whether they still want to use an accelerator when they have all the details.

Before I explain why this is the case, let me point out a few caveats:

The only SSL accelerator that we officially support with the Directory Server is the Sun CryptoAccelerator 1000 on SPARC-based Solaris systems using the 32-bit version of the server. Even though the NSS library uses the standard PKCS#11 interface and we have tested with other accelerators (including the CryptoAccelerator 4000 and the NCP built into the UltraSPARC T1 processor on Sun Fire T2000 systems) the SCA 1000 is the only one that is officially supported at this time.
There are known issues when attempting to use PKCS#11 devices with the Directory Server on Solaris 10. These issues are being addressed for the upcoming Directory Server 6 release, although there are workarounds for the current version if necessary.
The information contained in this post is not specific to the Sun Directory Server, but applies to any case in which LDAP is used over SSL. Any LDAP directory server that can be used in conjunction with an SSL accelerator would see the same results (unless they've got a broken SSL implementation to start with).
The information contained in this post is not specific to Sun's SSL accelerators, but applies to any SSL accelerator currently available, including specialized network devices like switches and load balancers. For many protocols (especially HTTP) Sun's accelerator boards provide huge performance improvements for SSL-based communication.

To understand why you can't really accelerate LDAP over SSL, you need to understand a little about how SSL works and then a little about how SSL accelerators work. This post is a feeble attempt to provide that information for you to understand why this is the case.

Many encryption algorithms fall into two categories: symmetric or asymmetric. Symmetric encryption algorithms, like 3DES, RC4, and AES, are those that use the same key to decrypt data that was used to encrypt it (so both parties must have the same key). Asymmetric encryption algorithms, like RSA and Diffie-Hellman, use different keys for encryption and decryption (often called public and private keys). Symmetric encryption is generally much faster than asymmetric, but secure key management is usually much easier with asymmetric algorithms. The beauty of SSL is that it combines both forms in order to get the convenience of asymmetric key management with the speed of symmetric encryption. It accomplishes this by generating an essentially random value to use as the key for the symmetric encryption, using asymmetric encryption to securely transfer that key to the other system, and then switching to the faster symmetric encryption mechanism using that randomly-generated key to actually encrypt the communication.

Because the expensive asymmetric encryption is all done up front when the connection is established, the longer that a connection can remain open and using the agreed-upon symmetric key, the smaller the relative cost of the asymmetric key exchange will be. Unfortunately, however, many protocols (like HTTP) use relatively short-lived connections in which the connection is established, the expensive asymmetric key exchange occurs, the request is sent using symmetric encryption, the response is read using symmetric encryption, and the connection is closed. If the connection had remained open and the client had used it for additional requests, then the relative expense of the key agreement up front would seem much smaller by comparison. Although using the same connection for lots of requests is very common in well-behaved LDAP clients, it's much less common with protocols like HTTP and so the expense seems greater.

I should point out that there is a notable exception to this for cases in which a client needs to create multiple SSL-based connections to the same server for separate requests over a relatively short period of time. In these instances, there is an optimization in the form of SSL session sharing that allows the client to perform the expensive asymmetric key exchange when establishing the first connection, but then having subsequent connections indicate that they want to use the same symmetric key as was used for the first. This is good and can really cut down on the expense of the asymmetric encryption, but only for cases of communication between the same client and the same server. It doesn't really provide any help for cases in which lots of different clients are establishing connections to the same server, or where a single client establishes lots of different connections to different servers, but nevertheless it can at least be of use in some cases (e.g., an HTTP client needs to use one connection to retrieve an HTML document and then other connections to retrieve associated files like images, stylesheets, or frames, or a user loads one page and then clicks a link on that page to load another from the same server).

The net effect of this is that the worst SSL-based performance will come from applications that need to handle lots of short-lived connections from a wide range of clients. This is exactly the kind of access pattern that most Web servers endure. On the flip side, the best SSL-based performance will come from applications that use a relatively small number of persistent connections, or failing that, those whose connections come from a relatively small number of clients such that session keys can be re-used. This is exactly the kind of access pattern to which most directory servers are subjected. Since the most common scenario (the one used by Web servers) is also the worst-performing, then this is the one that is targeted by SSL accelerators. Such accelerators have special hardware and built-in logic that allows them to process the expensive calculations associated with asymmetric encryption (particularly for algorithms like RSA). Since this is a very rare access pattern for directory servers, they will generally see little to no benefit from this optimization.

Some SSL accelerators, like the Sun CryptoAccelerator 4000, also make an attempt to speed up certain symmetric encryption mechansisms (also called bulk ciphers), which most commonly includes algorithms like 3DES, and AES. These algorithms are among the most expensive of the symmetric mechanisms, with 3DES taking about 108 clock cycles per byte of encrypted data (source) and 128-bit Rijndael (used by AES) somewhere around 291 clock cycles per byte (source). On the other end of the spectrum is RC4, which only requires about 7 clock cycles per byte. In fact, RC4 is so fast on its own that it's virtually impossible to accelerate through any means outside of the CPU itself because the expense of transferring the data to another board and the associated interrupt handling would negate any potential gains that could be had through specialized hardware. About the only case where one might see improvement would be if the CPU itself had special enhancements for recognizing and optimizing these kinds of operations and none of the major CPUs on the market have this capability (yet).

In many SSL implementations, including the one used by our Directory Server, RC4 is used as the preferred cipher for bulk encryption. This means that short of RC4 acceleration in the CPU itself, you're not going to see this symmetric encryption get any faster than it already is. While it is possible to configure the server to use other ciphers like 3DES and AES that can be accelerated, it doesn't make sense to do so from a performance perspective because even the accelerated forms of these algorithms are slower than unaccelerated RC4.

Another factor that impacts the ability to accelerate symmetric encryption in SSL is the amount of data that needs to be encrypted. The more data there is to encrypt, the bigger benefit that you will see from accelerating it (assuming that the cipher being used can be accelerated). Things like HTML documents and images that are commonly transferred over HTTP are usually at least several kilobytes in size while most LDAP messages top out around a couple hundred bytes, although this will obviously vary with the type of operation (e.g., the size of search result entries depends on the size of the entries and what subset of the attributes are included). So unless your LDAP clients are routinely performing searches that match large numbers of entries and/or those searches are returning entries with lots of data, then there isn't a whole lot of room for improvement by accelerating the symmetric encryption.

So as you can see, it turns out that LDAP is pretty much a poster child for SSL in the way that clients tend to avoid the expensive asymmetric calculations and use small message sizes, and in most cases the performance hit from using SSL in the Directory Server is smaller than you might have expected. Unfortunately, the same qualities that make it a good match for software-based SSL make it the ugly stepchild when it comes to accelerators because the things that they do to help HTTP don't help LDAP. As usual, let me point out that there are exceptions to every rule and I'm sure that there are some customers out there that do have directory access patterns that would lend themselves to acceleration, and in fact I have run several tests with LDAP over SSL using contrived workloads in which I have seen performance improvements resulting from hardware accelerators, but it's just not very common. And of course I should also note that there may be other reasons to use one of these boards than just to speed up SSL (e.g., secure key storage is an attractive feature of the Sun CryptoAccelerator 4000, particularly with its FIPS 140-2 level 3 certification). But if you're in the market for an accelerator in an attempt to speed up LDAP over SSL, then you might want to take the time to investigate this more closely to see if it would really help in your environment. Perhaps a trip to an iForce Customer Benchmarking Center might be in order.

Posted by cn_equals_directory_manager ( Jan 03 2006, 09:30:39 AM CST ) Permalink Comments [5]