cn=Directory Manager

cn=Directory Manager
All about Directory Server

All | Personal | Sun

« Previous month (May 2006) | Main | Next month (Jul 2006) »

Monday July 31, 2006

Posts now mirrored at www.directorymanager.org

I've felt for a long time that the information here is too hard to find once it gets pushed off the front page, and a few others have confirmed that for me. I've been meaning to mirror this content somewhere else for a long time, but it wasn't until this weekend that the time to get it done and the ambition to do so.

Now you can go to http://www.directorymanager.org/blogs/ to see a list of all of the posts that I've made here in the past. As I add more content in the future, I'll try to keep them in sync.

There's not much point to trying to look around the rest of the site, since there's nothing else there yet. I may get around to adding more stuff in the future, but for now the mirror of this content is all that there is.

Posted by cn_equals_directory_manager ( Jul 31 2006, 08:30:09 AM CDT ) Permalink Comments [2]

Friday July 28, 2006

Understanding the Backend Monitor Entries

Earlier, I wrote about the cn=monitor entry and the kinds of information that it provides. I mentioned in passing that the backendMonitorDN attribute specifies the DNs of the monitor entries for the backends defined in the server, but I didn't go into any detail on what data they provided. There is a lot of useful information in the backend monitor entries, and they deserve their own post to describe their content.

The backend monitor entries have a DN of "cn=monitor,cn={backendID},cn=ldbm database,cn=plugins,cn=config", where "{backendID}" is the name of the backend you're interested in monitoring (e.g., "userRoot" is the name for the default backend when a new instance is created). The monitor entry provides information specific to that backend, including cache and database statistics. The attributes in these entries include:

currentEntryCacheCount -- This specifies the number of entries that are currently held in the entry cache for the backend.
currentEntryCacheSize -- This specifies the amount of memory (in bytes) that the server believes the backend's entry cache is consuming. This number may be somewhat inaccurate (see my earlier post on cache sizing for why this is the case), although the accuracy should be improved in the upcoming Directory Server 6 release.
database -- This specifies the type of database that is in use for the backend. This should always be "ldbm database".
dbEntryCount -- This specifies the number of entries in the backend database (i.e., the number of unique keys in the id2entry database file).
dbFileCacheHit-# -- This specifies the number of times the server requested an entry from the associated database file and the data was already contained in the database cache.
dbFileCacheMiss-# -- This specifies the number of times the server requested an entry from the associated database file and the data was not contained in the database cache.
dbFileName-# -- This specifies the path to a database file for the backend. Note that the "-#" part of this and other dbFile* attribute names actually indicates an integer value. Each database file accessed by the server will be given a different number, and the same number will be used for the DB file name, hit and miss count, and page in and out count so that those statistics can be associated with the appropriate database file.
dbFilePageIn-# -- This specifies the number of times that content from the associated database file had to be paged into the cache.
dbFilePageOut-# -- This specifies the number of times that content from the associated database file was paged out of the cache (presumably to make room in the cache for another database page that needed to be paged in).
entryCacheHitRatio -- This specifies the percentage of the time that a request to retrieve an entry from the entry cache actually finds that entry in the cache. It will be equal to 100 * entryCacheHits / entryCacheTries.
entryCacheHits -- This specifies the total number of times that a request to retrieve an entry from the entry cache actually finds that entry in the cache.
entryCacheTries -- This specifies the total number of times that a request was made to retrieve an entry from the entry cache, regardless of whether or not the target entry was in the cache.
ldapEntryCount -- This specifies the number of LDAP entries in the backend database. The difference between this and the dbEntryCount attribute is that the ldapEntryCount does not include special entries like tombstones that are not generally accessible over LDAP.
maxEntryCacheCount -- This specifies the maximum number of entries that may be held in the entry cache, or -1 if there is no upper limit to the number of entries to store.
maxEntryCacheSize -- This specifies the maximum amount of memory in bytes that the server will try to use for the entry cache. Again, this may be a little inaccurate, but Directory Server 6 should be better than 5.2 in this area.
nextID -- This specifies the entry ID that will be used for the next entry that is added to the backend database. This is generally not that useful for monitoring, but may be helpful for some corner cases when debugging problems.
readOnly -- This indicates whether the backend is configured to be read-only. Note that even if the backend itself is not configured in read-only mode, the server-wide read-only configuration may be enabled which has the same effect but across all backends.

Posted by cn_equals_directory_manager ( Jul 28 2006, 05:29:08 PM CDT ) Permalink

Thursday July 27, 2006

Understanding the cn=monitor entry

The Directory Server's "cn=monitor" entry provides a means of gaining insight into what's going on inside the server. However, there may be attributes in this entry with names that don't clearly explain what kind of information they provide. In this post, I'll try to shed some light on them.

The attributes contained in this entry include:

backendMonitorDN -- This specifies the DNs of the monitor entries for each of the backends in the server. The backend monitor entries provide details about the server's interaction with the backend databases. The contents of these entries could easily constitute a separate post.
bytesSent -- This specifies the total number of data bytes included in the responses that it has sent to clients. Note that this only includes data bytes and does not include packet headers (e.g., link-level, IP, and TCP headers).
cache-avail-bytes -- This specifies the amount of free memory in the system that could possibly be using for caching. Note that it does *not* specify how much of the cache is unused like one might think. Honestly, it's not all that useful.
connection -- This is a multivalued attribute with a separate line per established connection. This attribute will only appear if the monitor entry is retrieved by the root DN (e.g., "cn=Directory Manager"). Values are constructed in a colon-delimited form with the following components:
1. The slot in the connection table, which generally equals the file descriptor number
2. A timestamp that specifies when the connection was established (in UTC)
3. The number of operations initiated on the connection
4. The number of operations completed on the connection
5. If the connection is in the process of reading data from the client, then the letter "r". Otherwise, it will be blank
6. The DN of the authenticated user for the connection
7. The protocol the client is using to communicate with the server
connectionPeak -- This specifies the maximum number of concurrent connections that have been established to the server at any given time.
currentConnections -- This specifies the number of connections that are currently established to the server.
currentTime -- This specifies the current time according to the system clock on the server (in UTC).
dTableSize -- This specifies the size of the file descriptor table for the server. This should be equal to the value of the nsslapd-maxdescriptors configuration attribute.
entriesSent -- This specifies the total number of search result entries sent to clients in response to search requests.
heapMaxHighHits -- This specifies the number of times that the server heap has grown large enough that it was necessary to free entries in the cache while holding an exclusive lock on the cache. Having this value increase is less desirable than having the heapMaxLowHits value increase, because it can have a significant impact on performance.
heapMaxLowHits -- This specifies the number of times that the server heap has grown large enough that it was necessary to free entries in the cache, but without the need to lock the entire cache. This will have a lesser impact on performance than cases in which it is necessary to hold the lock while freeing entries.
nBackends -- This specifies the number of backends (also called databases) configured in the server.
opsCompleted -- This specifies the total number of operations that have completed processing since the server started. The difference between opsInitiated and opsCompleted should equal the number of operations in progress.
opsInitiated -- This specifies the total number of operations that have been initiated since the server started.
readWaiters -- This specifies the total number of client connections for which the Directory Server is waiting to read more data from the client (i.e., the client has started sending a request but has not yet completed it). This should be equal to the number of connections with the "r" notation in the connection attribute.
request-que-backlog -- This specifies the number of requests that are currently sitting in the work queue waiting to be picked up by a worker thread. Having more than a couple of requests sitting in the work queue at any given time generally isn't a good thing because it means that all of the worker threads are busy processing other requests so new requests have to wait.
startTime -- This specifies the time that the server was started (in UTC time).
threads -- This specifies the number of worker threads currently configured for use in the server (as controlled by the nsslapd-threadnumber configuration attribute).
version -- This is the Directory Server version string. It can be altered to an extent via the nsslapd-versionstring configuration attribute to help it deal with clients that foolishly are hard-coded to only work with a specific server version, but in general changing that string is not recommended.

I'd say that the most important thing to watch here is the request-que-backlog attribute, since the higher it is the more requests are having to wait in the queue before they can be processed. If you consistently see currentConnections growing over time, then that could indicate that one or more of your clients aren't properly closing their connections when they're done with them, and if it gets too close to the dTableSize value (and in particular, dTableSize minus the value of nsslapd-reserveDescriptors) then the server could start rejecting new connections.

Posted by cn_equals_directory_manager ( Jul 27 2006, 08:54:32 AM CDT ) Permalink Comments [3]

Wednesday July 26, 2006

Setting Up and Using SSL

On a personal note, things have been really busy lately. In fact, they've been so busy that I let a couple of comments on my last post go unanswered for far too long. There were some meaty issues in those questions, so to make up for it I'll try to provide more detailed information in separate posts spread over the rest of the week. The one I've got slated for Friday looks to be a real doozy, so you'll definitely want to tune in for it.

The first of these questions that I'm going to address is about how to configure SSL in the Directory Server, and in particular using and testing SSL client authentication in which the client is able to authenticate to the server using a certificate (via SASL EXTERNAL) rather than the more traditional DN and password used for simple authentication.

I would be remiss if I didn't first point out that our publicly-available documentation does address these issues. The Directory Server Administration Guide does have sections that cover both enabling SSL in the server and in the clients. However, while the documentation does provide some coverage in this area, there may be a few pieces to the puzzle that aren't included, and others that could be clarified.

I've done quite a bit of SSL-related work in the server, and this isn't the first time that I've heard these questions. In fact, I've written a few documents on the subject to help various customers with different issues. I've taken one that most closely matches the original request (and probably the one that provides the most complete picture of setting up and testing SSL with server and client authentication), spruced it up a bit, and made it available here.

Note that this document really deals with the full course of configuring SSL in the server, from getting the certificates (either self-signed or requested from an external authority), enabling SSL, and testing it with ldapsearch. It also heavily favors the use of the command line instead of the graphical administration console, and some people prefer that over having to launch the administration console just for a couple of clicks (I know I do). Some of the steps may change in the upcoming Directory Server 6.0 release, but the command-line tools coming in that release will be a welcome sight to a great many administrators, and they will provide ways of accomplishing most of the things covered in my document.

Posted by cn_equals_directory_manager ( Jul 26 2006, 08:39:53 AM CDT ) Permalink

Thursday July 20, 2006

The Directory Server Access Log Format

The Directory Server access log can be one of the most useful tools for understanding client behavior and debugging problems when things aren't working as expected. There's a lot of information that can show up in the log, and it may not be obvious in all cases what all of the information means so I'll give it a quick summary here.

Common Log Elements

All access log file lines have a few common elements:

{timestamp} -- The time that the message was logged, including the date, time, and offset from GMT.
conn -- The connection ID for the connection that triggered the log message. All log elements for actions associated with a given client connection will have the same connection ID. If you see a connection ID of -1, then it indicates that the associated message is associated with an internal operation and was not requested by an external client.
op -- The operation ID for the client operation associated with the log message. This can be used in conjunction with the connection ID to identify operations for a given client (the response log message for an operation will have the same conn and op values as the corresponding request log message). If you see an operation ID of -1, then it indicates that the log message was not associated with an LDAP request from the client but rather was some other notable event like establishing or terminating the connection.
msgId -- The message ID from the LDAP message that the client used in the request to the server. This can be useful in helping to debug the protocol-level communication with the client. As with the operation ID, a message ID of -1 indicates that the log message is not associated with a protocol operation requested by the client but some other event.

Connection Establishment

When a new connection is established, you'll see a line with the following elements:

fd -- The file descriptor number for the socket associated with the client connection.
slot -- The slot in the internal connection table for the client connection structure. This is usually the same as the file descriptor number.
{protocol} -- The protocol that the client is using to communicate with the server. This could be LDAP, LDAPS, or HTTP (for DSML).
connection from {address} -- The IP address of the client (as seen by the server, so this could be the address of an intermediate system like a proxy)
to {address} -- The IP address on the server system to which the connection was established, which can be useful if the server is multi-homed

If it's an SSL=based connection, then you'll also see a line indicating the cipher that was used, like "SSL 128-bit RC4". If the client sent its own certificate to the server, then the subject of that certificate will also be included.

Connection Closure

When a connection is closed, you'll generally see a couple of lines. The first one indicates that the connection is going to be closed and the second that it has been closed. The first of these two lines also contains a closure code that can help indicate why the connection was closed. These closure codes include:

A1 -- The client aborted the connection (i.e., closed without sending an unbind). This is generally only seen on Windows systems, as B1 usually means this on UNIX systems.
B1 -- On UNIX systems, this virtually always means that the client closed the connection without sending an unbind request. It could also mean that the client sent a malformed request to the server.
B2 -- The client sent a request that was too large (i.e., exceeded the max BER size).
B4 -- The server got an error when it was trying to send data to the cient.
T1 -- The client connection was closed because it remained idle for too long (i.e., the idletimeout was reached).
T2 -- The server closed the client connection because the client started to send a request but took to long to send the rest of it, or because the server started to send a response to the client but the write blocked for too long (i.e., the ioblocktimeout was reached).
P1 -- The client connection was closed by a plugin.
U1 -- The client connection was closed when the client sent an unbind request.

Response Lines

Whenever the server processes a request from the client, the server will generally log two lines: one when it starts to process the request, and a second when that processing has completed and the server has sent the response (unless there isn't a response, like with an unbind or abandon). The request line may vary based on the type of operation, but the response line will always have the following elements:

err -- The LDAP result code for the operation. A result code of zero generally indicates that the operation was successful, while nonzero indicates an error (although there are a few exceptions, like 5=compare false, 6=compare true, or 14=SASL bind in progress).
tag -- The decimal value of the BER type tag from the protocol op of the response message. They include: 97=bind, 101=search, 103=modify, 105=add, 107=delete, 109=modify DN, 111=compare, 120=extended operation.
nentries -- The number of entries returned to the client. This will only ever be nonzero for a search, but it's included on all response lines.
etime -- The length of time required to process the request, in seconds (although it's possible to make it sub-second by changing nsslapd-accesslog-level to 131328 in cn=config).

Bind Requests

The following elements may be logged as part of a bind request:

dn -- The DN contained in the bind request. Note that this may be empty for anonymous binds, but it will also often be empty for SASL binds since the identity of the user is defined in another manner.
method -- The authentication method. It will be "128" for a simple bind, or "sasl" for a SASL bind.
version -- The LDAP protocol version. It will be 3 for LDAPv3 or 2 for LDAPv2.
mech -- The SASL mechanism used (only present in SASL binds).

Note that the response line for a bind operation will also contain an additional field "dn", which is the identity of the user that actually authenticated. This can be useful for cases in which the client used SASL authentication so the bind DN isn't visible in the request line, and it's also a good way to double-check the authentication performed since there can be cases in which a successful bind can be authenticated as a user different from the one specified in the DN field of the bind request.

Search Requests

The following elements may be logged as part of a search request:

base -- The base DN for the search request.
scope -- The scope for the search request. A value of 0 indicates that it's a base-level search, 1 means single-level, and 2 means whole subtree.
filter -- The string representation of the filter from the search request.
attrs -- The list of attributes requested by the client, or "ALL" if the client didn't request any specific attributes.

Add, Delete, and Modify Requests

Add, delete, and modify requests all have the same format for the request log messages. It contains only the DN of the entry being added, deleted, or modified.

Modify DN Requests

The following elements may be logged as part of a modify DN request:

dn -- The original DN of the entry being moved/renamed.
newrdn -- The new RDN for the entry.
newsuperior -- The new superior DN for the entry, or "(null)" if the newSuperior element was not included in the request.

Compare Requests

The following elements may be logged as part of a compare request:

dn -- The DN of the entry targeted by the compare request.
attr -- The attribute for which the comparison is being made.

Extended Operation Requests

Extended operation requests have only a single element in the request line, which is the OID for the extended operation.

Posted by cn_equals_directory_manager ( Jul 20 2006, 08:05:27 AM CDT ) Permalink Comments [3]