Wednesday March 29, 2006 | cn=Directory Manager All about Directory Server |
Understanding Indexing and the ALLIDs ThresholdProper index configuration is a very important factor in the performance of the server. If you're missing a key index, then searches that should take microseconds can take hours on large data sets. Conversely, having too many indexes can adversely impact the performance of write operations, particularly for adds and deletes, because it requires more information to be updated in the database. And for both reads and writes, having a poor index configuration (e.g., a bad ALLIDs threshold) can also contribute to sub-optimal performance.In order to understand how to optimally configure indexes in the Directory Server, it is necessary to have at least a basic understanding of how they really work. For the purposes of this discussion, we'll focus on the three most popular index types: presence, equality, and substring. Approximate and extensible matching indexes work in a manner similar to equality indexes. VLV indexing is very different from attribute indexing, but it's also not as relevant to this discussion, so I'll leave it alone for now. The Directory Server database is essentially a set of B-Trees, which provide ordered mappings between key-value pairs (much like the TreeMap structures in Java) that scale extremely well without adversely impacting performance. Given a key, the corresponding value can be retrieved very quickly. In each Directory Server backend, one B-Tree database is used to hold all of the entry data. Each entry is assigned an integer value (the entry ID), and given that entry ID it is a very fast operation to retrieve the corresponding entry data. All of the other databases are indexes, which are used to define an association between the index keys and the entry IDs for the entries that correspond to those keys. Most Directory Server indexes are used to map attribute contents to entries, and we'll get into that in more detail later. But there are some special system indexes that define other kinds of mappings that are very important to the operation of the server. The most notable ones include:
Presence attribute indexes are used to keep track of all entries with a given attribute. For each attribute with a presence index, there is a single key (the plus sign, "+"), whose value is a list of all the entry IDs for all entries that contain at least one value for that attribute. When the server is processing a search like "(sn=*)", it first retrieves the "+" key from the sn index database to get a list of all the entry IDs and then goes to id2entry in order to get the entries with those IDs. Equality indexes are used to keep track of all entries with a given attribute value. Each equality index key is the normalized form of the value prefaced by the equal sign, and the value is a list of all entries that contain that particular attribute value. So when the server is processing a search like "(sn=Smith)" it will retrieve the ID list for the "=smith" key from the sn index and then will go to id2entry to retrieve the entries for those keys. Substring indexes are used to keep track of all entries with attribute values containing substrings. Each value is normalized and then broken into unique three-character substrings, with the beginning of the value signified by the carat symbol ("^") and the end of the value denoted by the dollar sign ("$"), and each of these substrings is prefaced by the asterisk ("*") to form the index keys. So the value "Smith" will result in five substring keys: "*^sm", "*smi", "*mit", "*ith", and "*th$". Whenever a substring search is received by the server, it may need to retrieve multiple keys and merge the ID lists together, depending on the length of the provided substring. As you can see, a single attribute value can result in several index keys. If an attribute has a substring index, then it can be particularly expensive to maintain because each value can have multiple index keys. The more index keys that are involved in a write operation, the more expensive that operation will be, so ideally substring indexes should be kept to a minimum. Another big factor in the performance impact of maintaining indexes is the number of entries matching each key. As the number of entries matching a given key increases, the cost of maintaining that particular index key increases. Further, some index keys may match a very large percentage of the entries in the server (e.g., the "=top" key for the objectClass index) and therefore could play a part in degraded performance for writes to a large percentage of the entries. However, this does not actually happen because of the ALLIDs threshold (as configured by the nsslapd-allidsthreshold attribute in the cn=config,cn=ldbm database,cn=plugins,cn=config entry). This configuration attribute controls the maximum number of entries that will be allowed to match a given index key. After an index key matches more than this number of entries, the value is replaced with a special token that indicates that this particular key will no longer be maintained. Other keys within the same index will still be maintained, so you could get into a case where "(sn=Smith)" is unindexed because the "=smith" key has hit ALLIDs while "(sn=Smithers)" would be indexed because the number of entries matching "=smithers" is below the limit. When configuring the Directory Server's ALLIDs threshold, there are generally four things to consider:
As with most things, the rules for tuning ALLIDs are a little fuzzy since there are always special circumstances that can have an impact in the decision. In most cases during our testing, we don't tune ALLIDs at all. Even if we're looking at hundreds of millions of entries, the default ALLIDs value is fine because all of the operations are such that none of the operations target multiple entries, so the indexes we'll be using will typically only have one ID per key. This is a very common real-world scenario, especially for the larger directories, since the clients accessing it are frequently Web applications that only need to perform each operation in the context of one user. We're actually more likely to increase ALLIDs for the smaller "enterprise" directories since they are the servers that need to serve as e-mail address books and telephone directories which do need to process searches that return multiple entries. Hopefully, having a basic understanding of the way that Directory Server indexing works will help make the problem clearer so you can make more informed decisions about how best to tune it. Posted by cn_equals_directory_manager ( Mar 29 2006, 10:30:02 AM CST ) Permalink Comments [5] Running Out of Anonymous PortsI've recently been asked from a few different people about a problem that's come up when they're doing performance testing and things just seem to "go to sleep" for some period of time. Upon closer investigation, we found that the problem was that the client was creating a separate connection per request and that the system was simply running out of anonymous connections to use for the requests.To explain what this means, it's important to understand how the OS keeps track of connections. Each TCP connection that a system knows about is uniquely identified by a set of four properties: the local address, local port, remote address, and remote port. The remote address and port are those of the target service that we're trying to access, and they're usually the only ones that we need to worry about or be aware of (although not in this case). The local address is the IP address on the client system that is being used to originate the connection to the server, and the local port is one that the system has dynamically allocated to distinguish it from other connections on the same system to the same remote host address and port. Because we usually don't care what the local port is, it can also be called an anonymous port. Each TCP connection that a system knows about must have a unique combination of these four properties. Most of the time, this isn't a problem, but if you're creating lots of connections to the same target system (e.g., if you're running a benchmark), then chances are pretty good that the remote address, remote port, and local address will be the same for all of those, and therefore the only thing left to distinguish all of those connections is that local "anonymous" port. The TCP port range is actually pretty limited: they only go from 1 to 65535, which means that you have a theoretical maximum of 65535 concurrent connections from the same remote address to the same remote address and port. However, the first 1024 of those are called privileged ports and are off limits unless you're the superuser, because they're typically meant to be used for ports associated with network services coming into the system rather than going out. And because the 1-1024 is reserved for root-only (or if you're on Solaris 10, users with the NET_PRIVADDR privilege), a lot of services that aren't meant to run as root use ports outside of that range for the same reason. To accomodate this, many systems don't start the anonymous port range until significantly higher. On Solaris, the default to restrict anonymous ports to the range between 32768 and 65535, so that means that there's only the ability to have 32768 concurrent connections to the same target system. Even if that value is smaller than you might have thought, you're probably still not approaching anywhere near that many concurrent connections with your testing. So why do you run out of anonymous ports when you're creating a lot of very short-lived connections? It's because whenever a connection is closed it doesn't really go away immediately, but rather it gets put into a state like TIME_WAIT or CLOSE_WAIT in case there's any unsent data that still needs to be handled. It stays in this state for a brief period of time (60 seconds by default on Solaris 10) until the system decides that the connection is really dead and that it's not going to be re-used, at which point it gets freed up so that port can be used by anonther connection if necessary. So how do you get around these problems? There are a number of possibilities:
SLAMD can help you with most of these. It uses persistent connections by default, although in most cases you can choose to always disconnect if you want to. Spreading the load across multipel clients is one of the primary reason that SLAMD was written, so it's pretty good at that. And after a recent commit, it always sets the SO_LINGER socket option, along with TCP_NODELAY and SO_REUSEADDR. I suppose that you could use the Exec Job to invoke ndd to decrease the lowest anonymous port, but that would probably be cheating. Posted by cn_equals_directory_manager ( Mar 23 2006, 09:15:45 AM CST ) Permalink Comments [8] Frequently-Asked Questions #5: Rejecting Anonymous BindsFrom time to time, we get questions about whether or not the Directory Server has the ability to reject anonymous bind requests. The short answer is that it does not, and that it's not likely to get that capability any time soon because it wouldn't actually do anything.To explain more completely, it is necessary to understand a little about how LDAP actually works. It's a TCP-based protocol that uses persistent connections. When a client establishes a connecton to the Directory Server, it is considered to be in an unauthenticated state. There is no need to send a bind request to the server before sending other operations, and any such requests that are sent will be processed with the set of permissions available to the anonymous user. As a result, there is no need to bind at all in order to send anonymous requests, so preventing anonymous binds won't do anything. Another point to consider is what action the server should actually take if it were to reject an anonymous bind request. The LDAP specification requires that the Directory Server automatically set a client connection to an unauthenticated state upon receiving a bind request. This is done to ensure that if the bind fails, the client remains unauthenticated. Therefore, a client connection would have exactly the same state if the anonymous bind is successful or if it is rejected (or for that matter, if a non-anonymous bind request is unsuccessful). This means that rejecting an anonymous bind request would be completely ineffective because it would still accomplish the intended result. Even though the Directory Server does not have a mechanism to prevent anonymous binds, this is not a security problem of any kind. It is still entirely possible to configure access controls to reject operations requested by unauthenticated clients, so having such clients connected to the server is not of any risk. Further, as I have pointed out in the past, there may be certain entries like the server schema and the root DSE that should be available to anonymous clients so that the can best understand how to interact with the server. There is one notable exception to this policy. Section 4.2.2 of RFC 2251 states that an anonymous bind is one that uses simple authentication with no password, and that the bind DN is also typically a zero-length string but that this is not a requirement. However, in the real world virtually all anonymous bind requests have both a zero-length DN and password. In fact, there have been more than a few poorly-designed applications that authenticate to the Directory Server that have suffered from a security hole as a result of allowing anonymous binds with a DN but no password (if the user enters a valid username with no password, and the application doesn't check to make sure that a password was given, then the corresponding bind attempt will succeed as an anonymous bind, but the application will think that it was performed with the given username and may afford the client any permissions associated with that user). Because binds containing a DN are potentially dangerous and are almost always either a mistake or an attempt to circumvent security, Directory Server 6 will include an option to reject such binds as a measure of protection against such applications that might otherwise suffer from this security hole. Posted by cn_equals_directory_manager ( Mar 21 2006, 08:52:15 AM CST ) Permalink SLAMD Gets Job GroupsEven though I am the primary developer for SLAMD, it's not my idea. Both the concept and the name came from Stephen Shoaff, who is currently Director of Engineering for Directory Services. I've heard him tell many a tale about the dark days of Directory Server benchmarking when tools were fairly primitive and required a lot of human interaction. Manually adjusting the number of threads to squeeze the most performance out of the server. Setting alarms to get up in the middle of the night in order to get the results of one test and kick off the next. In the snow, uphill both ways (okay, maybe not that last one).When SLAMD came into the picture, things got a lot better. You can just schedule a few jobs and optimizing jobs, making each dependent on the one before it, and SLAMD wil take care of all the processing and you can come back several hours later (or if you want, whenever SLAMD sends you an e-mail after everything is done) to see the results. Whenever we get a new piece of hardware in our lab or a new build of the Directory Server, or need to do some other kind of testing, it takes us maybe ten to twenty minutes to schedule all the jobs and then we can come back the next day to see how it did (of course, checking in on things from time to time because we're impatient). But how quickly we forget the way things used to be. When you do it over and over again, that ten or twenty minutes spent scheduling twenty or thirty hours of work starts to seem a little more tedious than in the past. Since we almost always schedule the same core set of jobs, it would be nice if we could just do it all at once. Ideally, we could group all of those jobs together, link all the common parameters to each other, and get it down to about the bare minimum of work. This is exactly what a job group does. As of the commit that I made last night, you can create a new job group, add one or more jobs and/or optimizing jobs to that group with dependencies between them, and schedule that group as a single entity. When you add the first job to the group, you're given the option for each of the job parameters to indicate whether you want to allow the user to specify its value as a job group parameter or whether you want to hard-code a fixed value that will always be used. Then when you add additional jobs, you still have both of those options, but you can also map a parameter for that job to one that's already been defined in the job group. For example, if you're doing a Directory Server benchmark, then you can have a single job group parameter that specifies the address of the Directory Server instance to test, and then all of the jobs in the job group will reference that one value so you only need to give it once. With this new capability, the time required to kick off a full suite of tests should go from ten or twenty minutes to one or two. Instead of nine forms of job parameters, there's only one. I can't imagine it getting a whole lot easier than that, but I'm sure that before long we'll think of something. Until then, there's still a fairly sizeable wish list in other areas to consider. Posted by cn_equals_directory_manager ( Mar 13 2006, 09:57:19 AM CST ) Permalink Comments [2] Directory Server Denial of Service ProtectionThe Directory Proxy Server (which is included in the Directory Server Enterprise Edition) offers a number of features that can help protect your Directory Server instances from malicious or poorly-designed client applications that might otherwise consume lots of resources or otherwise interfere with the most efficient operation of the Directory Server. However, there are a lot of features built into the Directory Server itself that can also help. If you don't feel a need to use DPS, or if you have cases in which some clients might directly access the server without going through a proxy, then it's probably a good idea to make use of one or more of these.Search Time Limit Inefficient searches can take a long time to process. Depending on the amount of data involved, along with other factors, it can range from a few milliseconds to several hours or possibly even longer. This is certainly something that you'll want to prevent in most cases, and one way to limit that is to define a time limit for search operations. This can be done by setting the nsslapd-timelimit attribute in the cn=config entry to the maximum length of time in seconds that you want searches to be allowed to take. Note that this is a global setting, and it will automatically apply for all users (except the root DN, who is exempt from this limit). However, if there are special users for which you want to have a higher or lower limit, you can achieve this by putting the nsTimeLimit operational attribute in that user's entry so that limit will be enforced rather than the global limit. You should also note that it is also possible to specify a time limit in the search request itself, and if there is a conflict between the value configured in the Directory Server and the value included in the search request then the lower of the two will be the one that is enforced. Search Result Size Limit Another way that you can prevent clients from consuming large amounts of server resources is to place a limit on the number of entries that may be returned from any given search request. This is called the size limit, and the server-wide limit may be configured using the nsslapd-sizelimit attribute of the cn=config entry. As with the time limit, the size limit can also be overridden on a per-user basis by putting the nsSizeLimit operational attribute in that user's entry and setting it to the size limit that should be used instead of the server-wide value. And also like the time limit, the size limit can be specified in the search request itself, and the lower of the requested and server-enforced limits will be used in the event of a conflict. Search Candidate Size Limit The search size limit is a good way to limit the number of entries that may be returned to the client for a given operation, but it isn't necessarily a panacea because it only applies to entries that are actually returned to the client. It doesn't take into account entries that the server has to examine in the course of processing the search but that aren't returned for some reason (e.g., the user doesn't have permission to see them, or the server identified them as candidates during the index processing phase but it turned out that they didn't really match the criteria). In some cases, this could be a lot of entries, and the server has to expend effort on each of them while processing the search. However, as a means of protecting against situations like this, the Directory Server does have a configuration attribute that will limit the number of entries that the server will "look through" in the context of processing a search, regardless of whether or not they are actually sent to the client. This is controlled on a server-wide basis by the nsslapd-lookthoughlimit attribute in the cn=config,cn=ldbm database,cn=plugins,cn=config entry, but can be overridden on a per-user basis with the nsLookThroughLimit attribute in the user's entry. Reject Unindexed Searches The size limit, time limit, and lookthrough limit settings listed above can help significantly limit the amount of processing effort that the server expends while processing inefficient searches. Some of the most inefficient searches are those for which the server does not have appropriate indexes defined and therefore must look at a potentially very large result set (e.g., all entries in the database) in order to identify which ones match the criteria provided by the client. If this occurs, then the size/time/lookthrough limits will treat the symptoms of the problem, but you can also treat the root of the problem by configuring the server to reject unindexed searches. This can be done at a per-database level using the nsslapd-require-index attribute. If this setting is turned on, then any search request for which the server cannot use the defined indexes to identify a finite candidate list will be rejected. It is important to note that this cannot be overridden on a per user basis, so it should not be used if you have specific clients that might have a legitimate need to perform unindexed searches from time to time (although in those cases, it might be a good idea to restrict them to a single server instance and have all others configured to reject unindexed searches). Idle Time Limit In most cases, a well-behaved client should maintain one or more persistent connections to the server that can be used repeatedly to send many requests to the server. Less efficient clients will establish a separate connection for each request, although this may be acceptable for clients that don't need to communicate with the server all that often (i.e., several minutes or hours between requests). But the worst kind of client is one that establishes connections and then forgets about them after a while, establishing new connections for subsequent reqeusts without closing the existing ones. This is bad because clients like this can evenutally consume all the available file descriptors, which will interfere with the server's ability to accept new connections. To deal with this problem, the best approach is to set an idle timelimit (in seconds) using the nsslapd-idletimeout configuration attribute in the cn=config entry. If a client connection remains established for longer than this period of time without issuing any requests, then the server will automatically close it. If you do have individual applications that have legitimate need to have connections that may be idle for long periods of time, then you can override this setting on a per-user basis (in this case the "user" would be the account that the client application uses to bind to the server) with the nsIdleTimeout operational attribute in the user's entry. Unresponsive Client Time Limit Sometimes, certain denial-of-service conditions can occur through no fault of the client. For example, if a network failure occurs, or if a poorly-designed firewall or other network device closes drops connections without properly informing either end of the connection, then the server can have no way of knowing that a network connection is invalid. If the connection was idle at the time, then this isn't really a problem since the server won't try to use it and it will eventually be closed either by the idle timeout or when the underlying OS discovers that the connection is no longer valid. However, if this happens while the server is in the process of sending data to the client, or if the client simply stops reading data from the server when there is a lot more information to be sent, then the server can enter a condition in which further attempts to write to the client will block (i.e., "hang") until the data is read. This will tie up the associated worker thread, making it unavailable to process other requests. To prevent that from becoming a significant problem, the I/O block timeout may be used. This is controlled by the nsslapd-ioblocktimeout configuration attribute in the cn=config entry, and its value is the maximum length of time in milliseconds that attempts to send data to a client should be allowed to block before giving up and terminating the client connection. The current default value of 1800000 milliseconds (30 minutes) is far too high to be effective, and in future releases the default will be lowered to 30 seconds. It is recommended that existing deployments also define a more reasonable limit (e.g., something in the 30-120 second range). Maximum Request Size Because of the way that LDAP requests are encoded, it is necessary for the server to be able to hold the entire request in memory when it is decoded. Although this is generally not a problem, it does introduce an opportunity for malicious clients to try to send huge requests to the Directory Server in order for it to allocate large amounts of memory and to increase network load. To protect against this, the nsslapd-maxBERSize configuration attribute of the cn=config entry may be used to specify the maximum request size in bytes that the Directory Server wil allow. Any attempt to send a larger request than this size will cause the connection to be terminated. The default setting for this configuration parameter is two megabytes, which will be sufficient for most cases, but may be too small for directories that need to store large entries (e.g., those with static groups consisting of hundreds of thousands of members or more, or those that need to store large blobs of data like the configuration directory used by SLAMD). Maximum Concurrent Operations per Client Connection Because LDAP uses an asynchronous communication mechanism, there is no requirement for a client to wait on the response for one request before sending another. It is entirely possible for the Directory Server to be concurrently processing multiple requests for the same client using separate worker threads. A somewhat evil client may attempt to exploit this behavior by sending lots of requests to the server in an attempt to tie up all the worker threads and/or interfere with the ability of the server to process requests from other clients. However, this can be restricted using the nsslapd-maxThreadsPerConn configuration attribute in the cn=config entry. The value of this attribute may be used to specify the maximum number of worker threads that may be used to process requests from a single client at any given time. If a client attempts to exceed this limit, then its requests will be sent to the back of the work queue and they won't be eligible for processing until other processing for other requests on the same connection has completed. Maximum Number of Persistent Searches The persistent search control can be a useful way to receive notification of changes to entries in the Directory Server, but it can also be somewhat expensive because the server will create a separate thread for each persistent search operation. In order to prevent this from becoming a problem, the server can limit the number of persistent searches that may be registered at any given time with the nsslapd-maxpsearch configuration attribute in cn=config. If the maximum number of persistent search operations are already registered against the server, then subsequent attempts to create new ones will be rejected. Disk Space Thresholds A somewhat indirect form of denial-of-service may occur if one or more of the disk volumes used to hold Directory Server information become full. If a disk containing database contents should become full, then certain database operations will fail. If a disk holding log files fills up, then it won't be possible to write any more log information about the operations being processed. Fortunately, there are a few configuration parameters that can help prevent this from becoming a problem. In particular, the nsslapd-disk-low-threshold attribute in cn=config,cn=ldbm database,cn=plugins,cn=config will prevent write operations by all users except the root DN if free disk space drops below the specified amount, and the nsslapd-disk-full-threshold attribute in the same entry specifies the point at which even write operations by the root DN will begin to be rejected. In addition, the access, error, and audit log rotation mechanisms may be configured to automatically remove old log files if the amount of free disk space drops below a configured threshold (e.g., for the access log, this is the nsslapd-accesslog-logMinFreeDiskSpace attribute in cn=config), and it is also possible to specify the maximum amount of space that each type of log may consume with configuration attributes like nsslapd-accesslog-maxLogSize for the maximum size of a single access log file and nsslapd-accesslog-logMaxDiskSpace for the total space consumed by all access logs. Posted by cn_equals_directory_manager ( Mar 06 2006, 08:15:15 AM CST ) Permalink Comments [3] |
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||