WebbBLOG
Michael Webb's Weblog

Directory Server Performance Tuning: A Quick Reference (hardly)

Purpose: A discussion and resource document for Sun Java Systems Directory Server Customers

Background:

Many of our customers have deployed Directory Services utilizing Sun’s Java Systems Directory Server version 5.2. The Directory Services provided LDAP as a Naming Service for different types of ldapclients.

Some deployments have experienced loss of service, degraded performance and other observed issues.

Customers experiencing performance issues should investigate whether the Directory Server’s default performance settings are viable for their deployment. Many time customers want “magic numbers” to various settings that relate to the performance of the Directory Server. The purpose of this document to identify those settings that can affect performance, and provide (without testing or referencing client connection behavior) an adjustment to those settings.

Section One: The Directory Server

It is quite important to iterate at the outset of this discussion the basic function of the Directory Server.

The SJS Directory Server, along with all other popular LDAP servers, is highly configurable due to the variety of client connection methods, be it from SDK’s, LDAP versions, Operating Systems and third party applications. Once again RFC’s and other Definitions offer many different possibilities.

It can be described that the Directory Server “responds to” as opposed to “controls” incoming connection requests, and that the client connection behavior is equally important as the Directory Server’s internal performance.

It is an accepted component of every major LDAP implementation that some sort of performance-based testing be conducted to deliver the best possible results. In most cases, the default settings should apply; however, no other component can have more financial, technical and business impact as properly (or poorly) executed testing. Further, testing is iterative and can (should) be a vital part of every change to an infrastructure, LDAP or otherwise.

Further, it should also be noted that the "L" in LDAP stands for "Lightweight." A lightweight application should not be designed for multiprocessor support, complex and intensive threading equations, or for memory and connection-intensive operations. If the customer deployment requires these types of operations, LDAP is not the first choice of protocol. Since the client and server operations are tightly controlled by the LDAP RFC, and not subject to loose interpretation by each product vendor, any performance issues should first be investigated by reviewing configuration and environment settings.

Section Two: Persistent Searches, not to be confused with Persistent Connections

Persistent Searches are defined by RFC 2026 as part of the LDAP RFC maintained by the IETF. The definition of, and rules applying to, persistent searches in LDAP are managed by this RFC and not by product vendors. See the Intended Uses within the RFC for a complete description.

As a caution, the RFC memoranda stated the following in the Implementation Considerations:

Each active Persistent Search request requires that an open TCP connection be maintained between an LDAP client and an LDAP server that might

not otherwise be kept open. Therefore, client implementors are

encouraged to avoid using Persistent Search for non-essential tasks and

to close idle LDAP connections as soon as practical.¹

An opposition to the RFC stated as well:

[...] Each active Persistent Search registration requires an open TCP connection. Even a single LDAP client with multiple active Persistent Searches will require as many open TCP connections as it has Persistent Searches. There are LDAP servers that have been tested with hundreds of connections; some may even support thousands of connections, but somewhere a practical number of simultaneous open connections "service ceiling" will be reached beyond which response times will be unacceptable. Dozens or hundreds of LDAP Persistent Search open TCP connections will adversely impact LDAP Server performance everywhere.²

Tim Howes, noted author on LDAP, added his comments:

Psearch is very focused on solving a very limited

problem space, best illustrated by the scenarios

described in section 6, "Intended Use" of the document.

The scaling concerns, which bear on the intended

use, are discussed in section 7, "Implementation

Considerations". If you think either of these sections

need additional text to make the applicability of

psearch more clear, we'd definitely be interested

in incorporating your suggestions.³

Novell Corporation offers a competitive “eDirectory” Directory Server. They offered the following guidelines on persistent searches:

The Persistent Search operation is memory- and connection-intensive for the LDAP server, as it not only needs to maintain an open TCP connection but also needs to maintain the search request associated with that connection. Every change that happens in the directory database needs to be evaluated against the search criteria of the Persistent Search client, and the client is notified of the change or not accordingly.

It has been observed that beyond 50 concurrent Persistent Search clients, the performance of eDirectory may suffer. Hence, it is recommended that the maximum concurrent LDAP Persistent Search connections be restricted to the 5 - 10 range via the ldapMaxPSearchConnections attribute on the LDAP Server object. Of course, these numbers are all subject to differences in server capabilities.⁴

Persistent Searches are handled by the nsslapd-maxpsearch property⁵:

Defines the maximum number of persistent searches that can be performed on the Directory Server. The persistent search mechanism provides an active channel through which entries that change (and information about the changes that occur) can be communicated. Because each persistent search operation uses one thread, limiting the number of simultaneous persistent searches prevents certain kinds of denial of service attacks.

Property

Value

Entry DN

cn=config

Valid Range

1 to maximum threadnumber

Default Value

30

Syntax

Integer

Example

nsslapd-maxpsearch: 30

It is quite apparent that a client implementation that generates a large number of persistent connections (in this reference, >50 ) to a single Directory Server may indicate that the LDAP protocol may have not been the correct transport. However, horizontal scaling using Directory Proxy Servers, or a LDAP Consumer tier, to spread the load may assist.

The best solution, from an LDAP implementation, would be to limit persistent searches.

Persistent Connections are controlled by TCP/IP; and in contrast, should be considered in order to improve connection performance inherent with the multitude of connections generated by ldap clients.

Section Three: Maximum Threads per connection

Maximum Threads per connection are handled by the nsslapd-maxthreadsperconn property⁶:

Defines the maximum number of threads that a connection should use. For normal operations where a client binds and performs only one or two operations before unbinding, you should use the default value. For situations where a client binds and simultaneously issues many requests, you should increase this value to allow each connection enough resources to perform all the operations.

Property

Value

Entry DN

cn=config

Valid Range

1 to maximum threadnumber

Default Value

5

Syntax

Integer

Example

nsslapd-maxthreadsperconn: 5

The Directory Server Resource Guide provides the following guidelines:

If you anticipate that some applications may perform many requests per connection, consider increasing this value from the default of 5, but do not increase it to more than 10. It is typically not advisable to specify more than 10 threads per connection. ⁷

It would be arguable that a clients that consistently request >10 threads per connection would adversely affect overall performance of the environment.

Section Four: TCP NODELAY

Network issues are addressed by the nsslapd-nagle property⁸:

When the value of this attribute is off, the TCP_NODELAY option is set so that LDAP responses (such as entries or result messages) are sent back to a client immediately. When the attribute is turned on, default TCP behavior applies. That is, the sending of data is delayed, in the hope that this will enable additional data to be grouped into one packet of the underlying network MTU size (typically 1500 bytes for Ethernet).

Property

Value

Entry DN

cn=config

Valid range

on | off

Default value

off

Syntax

DirectoryString

Example

nsslapd-nagle: off

Since it is essentially an “either-or” test, it could be easily tested to see if it could improve performance.

Section Five: Startup Operation Threads.

When the Directory Server binaries start up, a number of threads are started. This is controlled by the nsslapd-threadnumber property⁹:

Defines the number of operation threads that the Directory Server will create during startup. The nsslapd-threadnumber value should be increased if you have many directory clients performing time-consuming operations such as add or modify. This ensures that there are other threads available for servicing short-lived operations such as simple searches.

Property

Value

Entry DN

cn=config

Valid Range

1 to the number of threads supported by your system

Default Value

30

Syntax

Integer

Example

nsslapd-threadnumber: 60

Use "request-que-backlog" in cn=monitor to determine if threads should be increased. request-que-backlog should be close to zero.

The Directory Server Tuning Guide provides the following information:

Consider adjusting the value of this attribute if any of the following are true:

- Client applications perform many simultaneous, time-consuming operations such as updates or complex searches.

- Directory Server supports many simultaneous client connections.

Multiprocessor systems can sustain larger thread pools than single processor systems. As a first estimate when optimizing the value of this attribute, use two times the number of processors or 20 + number of simultaneous updates.

Consider also adjusting the maximum number of threads per client connection, nsslapd-maxthreadsperconn, as discussed in Table 6-1. The maximum number of these threads handling client connections cannot exceed the maximum number of file descriptors available on the system. In some cases, it may prove useful to reduce, rather than increase, the value of this attribute.

Verify estimates through empirical testing. Results depend not only on the particular deployment situation but also on the underlying system.¹⁰

Section Six: File Descriptors

The preceding section indicated that nsslapd-threadnumber and nsslapd-maxthreadsperconnection properties are affected by file descriptors. Even though the connection threads are recommended to be be no more than 10, an aggregate of both properties should be reflected by corresponding values in the Directory Server.

File Descriptors are handled by the nsslapd-reservedescriptors property¹¹:

Sets the number of file descriptors Directory Server maintains to manage indexing, replication and other internal processing. Such file descriptors become unavailable to handle client connections.

Consider increasing the value of this attribute from the default of 64 if all of the following are true.

- Directory Server replicates to more than 10 consumers or Directory Server maintains more than 30 index files.

- Directory Server handles a large number of client connections.

- Messages in the error log suggest Directory Server is running out of file descriptors for operations not related to client connections.

Notice that as the number of reserved file descriptors increases, the number of file descriptors available to handle client connections decreases. If you increase the value of this attribute, consider increasing the number of file descriptors available on the system, and increasing the value of nsslapd-maxdescriptors.

If you decide to change this attribute, for a first estimate of the number of file descriptors to reserve, try setting the value of nsslapd-reservedescriptors to:

20 + 4 * (number of databases) + (total number of indexes) + (value of nsoperationconnectionslimit) * (number of chaining backends) + ReplDescriptors + PTADescriptors + SSLDescriptors

Where ReplDescriptors = number of supplier replica + 8 if replication is used, PTADescriptors is 3 if the Pass Through Authentication (PTA) plug-in is enabled (0 otherwise), and SSLDescriptors is 5 if SSL is used (0 otherwise).

The number of databases is the same as the number of suffixes for the instance, unless the instance is configured to use more than one database per suffix.

Verify estimates through empirical testing.

Section Seven: Suggestions from the Tuning Guide

The Tuning Guide referenced in the discussion devotes a majority of its material to some other equally important concepts and suggestions. Since they are a compilation of Best Practices and known results, they are viable options and must be explored and tested in conjunction with all Sun support efforts.

Searching and Sorting

For efficiency in searching and sorting, bear in mind the following: ·

- Avoid subtree searches that match a large number of entries. To retrieve one entry only, do the following:·

- If you are using the LDAP JDK, call the LDAPConnection.read() method, not the LDAPConnection.search() method.·

- If you are using JNDI, call the Context.lookup() method, not the DirContext.search() method.

Use multivalue attributes sparingly. If you must use them, for example, in the case of static groups, minimize the number of values in a multivalued attribute. And when updating multivalue attributes, delete and add specific values instead of replacing all the values with new ones.

In large databases, establish unindexed searches from searches on indexed attributes.

For searches and sorts, ensure that you correctly set the limits for the following attributes:

nsslapd-sizelimit -- This attribute specifies the maximum number of entries to return from a search operation. As soon as this limit is reached, the server returns an error message: Exceeded size limit error. The default value is 2000. A value of -1 means no limit, which we do not recommend.

nsslapd-lookthroughlimit -- This attribute specifies the maximum number of entries that Directory Server checks in response to a search request. If you bind as the Directory Manager, unlimited is the default and overrides other settings. However, if you set a a value for the operational attribute nsLookThroughlimit in the entry for binding, that value overrides the default. A value of -1 means no limit, which we do not recommend.

nsslapd-allidsthreshold -- The value for this attribute typically falls within a range of approximately 5 percent of the total number of entries in the directory. Since a change to this value requires a rebuild of all the indexes, you should set it before populating the database. For efficiency, avoid setting a high value (above 50,000) for this attribute--not even for large deployments.

nsslapd-timelimit -- This attribute specifies the search time limit for the database link in seconds. The default is 3600. A value of -1 means no limit, which we do not recommend.

To avoid impacting the server's performance, use server-side sorting only when absolutely necessary. Server-side sorting requires specific indexing in case of a large numbers of entries and incurs extra cycles. Likewise, to optimize performance, avoid persistent searches.

Server-side sorting can benefit from virtual list view (VLV) indexes. If a sorted search must occur regularly, add an appropriate VLV index.

Indexing and Partitioning

Here are our recommendations for indexing and partitioning:

A sound indexing strategy enhances performance because indexes accelerate searches. An index contains a list of values, each associated with a list of corresponding identifiers. By referencing the identifiers, Directory Server can quickly look up entries. Otherwise, Directory Server may have to check every entry in a suffix to find the matches for a search.

Index tuning for a Directory Server instance means maintaining only those indexes for which the benefits from faster search processing offset the costs incurred in processing updates and acquiring additional space. Maintaining useful indexes is an excellent practice. Conversely, maintaining unused indexes for attributes on which clients rarely search is a waste.

Because Directory Server aims at providing fast responses to read operations (queries), it is not a good choice for storing data that change frequently, such as inventory counts, purchase orders, and performance metrics. To achieve its goal, Directory Server depends heavily on indexing and storing data in memory cache. If the read-to-write ratio of Directory Server is low, the server's performance eventually becomes limited to the capacity of the master server.

Acknowledgment

Part of the content in this document originated in the publication Sun ONE Directory Server 5.x Best Practices and Guidelines written by the Directory Server Team in Sun Microsystems France. Our appreciation and thanks to that team.

Thanks also to Neil Wilson at Sun for his input.

The Sun Java System Identity Server Adoption Team at Market Development Engineering at Sun collaborates with independent software vendors in developing and integrating applications with Directory Server, Identity Server, and Portal Server.

Marina Sum is a staff writer for Sun Developer Network. She has been writing for Sun for 15 years, mostly in the technical arena.¹²

Summary

The SJS Directory Server manages several properties for performance, of which the following were reviewed:

nsslapd-maxpsearch

Default Value: 30

Maximum Recommended Value: 50

nsslapd-maxthreadsperconn

Default Value: 5

Maximum Recommended Value: 10

nsslapd-nagle

Default Value: off

Maximum Recommended Value: off

nsslapd-threadnumber

Default Value: 30

Maximum Recommended Value: 30 or (two times the number of processors or 20 + number of simultaneous updates), whichever is greater.

nsslapd-reservedescriptors

Default Value: 64

Maximum Recommended Value: 64 or

(

20 + 4 * (number of databases) + (total number of indexes) +

(value of nsoperationconnectionslimit)

* (number of chaining backends) + ReplDescriptors + PTADescriptors + SSLDescriptors

Where ReplDescriptors = number of supplier replica + 8 if replication is used, PTADescriptors is 3 if the Pass Through Authentication (PTA) plug-in is enabled (0 otherwise), and SSLDescriptors is 5 if SSL is used (0 otherwise).

)

nsslapd-sizelimit

Default Value: 2000

Maximum Recommended Value: 2000 or number of entries in database

(operational) nsslapd-lookthroughlimit

Default Value: not set

Maximum Recommended Value: 2000 or number of entries in database

nsslapd-allidsthreshold

Default Value: Not Recorded

Maximum Recommended Value: ~ 5 percent of total entries in the database, but < 50,000

nsslapd-timelimit

Default Value: 3600

Maximum Recommended Value: empirical based on testing, never unlimited.

It is the author’s perception that many of our customer’s environment does not normally exceed any of the default values for the referenced settings and that other factors would contribute to the performance and/or require any changes. Those factors should be addressed at the source as a first effort, and changes to the Directory Server settings should only be considered as the final alternative.

Finally, there is no substitute for properly executed and documented testing at any phase of the implementation.

References

The following references are public domain documents published on the Web. They were found by entering the search string "ldap client thread" at www.google.com.

Specific searches based on attributes were also conducted at Google.

None of these documents are proprietary, and some are provided with the Directory Server. Their collection and resulting commentary within this discussion represents proprietary work product provided by Sun Microsystems, Inc.

1. "Persistent Search: A Simple LDAP Change Notification Mechanism" (RFC 2026). http://www.mozilla.org/directory/ietf-docs/draft-smith-psearch-ldap-01.txt

2. "To: ietf-ldapext@netscape.com: Subject: Objections to draft-ietf-ldapext-psearch-01.txt: From: "Ed Reed, Scott Isaacson, Dave Huntbach (Ed Reed)" <ED#032#REED@novell.com>" http://www.openldap.org/lists/ietf-ldapext/199808/msg00129.html

3. "To: "Ed Reed, Scott Isaacson, Dave Huntbach (Ed Reed)" <ED#032#REED@novell.com>:Subject: Re: Objections to draft-ietf-ldapext-psearch-01.txt: From: Tim Howes <howes@netscape.com> " http://www.openldap.org/lists/ietf-ldapext/199808/msg00130.html

4. "Understanding and Using Persistent Search in Novell eDirectory" http://developer.novell.com/research/appnotes/2003/february/04/a0302047.htm

5. Directory Server Reference Guide provided as documentation for the Directory Server

6. Ibid.

7. "Sun Java(TM) System Directory Server 5 2004Q2 Performance Tuning Guide: Chapter 6: Managing Use of Other Resources " http://docs.sun.com/source/817-5220/servtune.html

8. Directory Server Reference Guide

9. Ibid.

10. "Sun Java(TM) System Directory Server 5 2004Q2 Performance Tuning Guide: Chapter 6: Managing Use of Other Resources " http://docs.sun.com/source/817-5220/servtune.html

11. Ibid.

12. Ibid.