Thursday May 25, 2006

Replication versus Cluster for DS High Availability

• Server Hardware Failure. In the event that a hardware failure leaves a system offline or in a degraded state, clients in a replicated environment could have instantaneous and transparent failover to another instance. A clustered environment can handle this failure as well, since the service would be migrated to another node, although there would be some downtime.
  
  
• Operating System Crash or Failure. With either replication or clustering, an OS crash is handled in essentially the same way as a hardware failure. If the OS hangs rather than crashes, then clients in a replicated environment would need to have some kind of timeout, after which the request can be redirected to another replica (the Directory Proxy Server can handle this transparently). In a clustered environment, an OS hang would be essentially treated in the same way as an OS crash.
  
  
• Storage Hardware Failure. If the storage hardware failure is catastrophic and results in the loss of all access to the data, then a replicated environment would be able to survive because each Directory Server instance has its own copy of the database on its own storage. In a clustered environment, it would not be possible to handle this because the nodes share the same storage. It is true that a failure like this is pretty unlikely if the storage itself has been designed with a highly-available configuration using RAID and redundant devices, but in that case a failure of one component could lead to degraded performance for some period of time (e.g., if RAID 5 needs to reconstruct data from parity for all reads, and there would be an I/O performance hit when rebuilding the failed component after it has been replaced).
  
  
• Whole Site Failure / Unavailability. If an entire site becomes unavailable for some reason (e.g., catastrophic network failure, extended power outage, natural disaster, etc.), then clients in a replicated environment can access other servers in other sites because replication works well in WAN environments. Clustering is a lot more complicated in WAN environments, and if it is even feasible in this kind of outage (e.g., when combined with replicated storage) it may result in a scenario in which multiple nodes are providing the service concurrently and there can be irreconcilable conflicts if multiple nodes allow data to be updated.
  
  
• Directory Server Crash. If the Directory Server process crashes, then clients in a replicated environment can be transparently redirected to other instances that are still online. In a clustered environment, there will be an outage until the Directory Server is restarted. In most cases it will be faster to restart the server on the same node than to migrate it to a different node, so the cluster doesn't provide much benefit there other than detecting the failure.
  
  
• Directory Server Hang / Freeze. If the Directory Server process hangs, then clients in a replicated environment can be transparently redirected to other instances that are still functional after some timeout is detected. Since the service itself is still available, the hung instance can be left in that state to see if it recovers itself (in most cases, this is simply a case in which all of the worker threads have been tied up processing expensive requests and the server will become responsive once processing on those requests has completed) or to capture diagnostic information like stack traces and/or core files that could help understand the cause of the hang. In a clustered environment, since there is only one instance it must be killed and restarted immediately to minimize the service outage, which prevents the opportunity for the server to recover itself, and which can inhibit the ability to identify the cause of the hang. In many cases, killing and restarting on the same system will be faster than killing and migrating to a different node. If the service does need to be forcefully killed, then it will also take additional time to restart as it may need to verify the integrity of the database.
  
  
• Directory Server Database Corruption. If the Directory Server database becomes corrupt for some reason in a replicated environment, then it will only impact one instance since all instances have their own database and replication is based on sending LDAP changes, not replicating at the database level. A clustered environment is not able to handle this kind of failure because it uses the same storage and database instance when failing over between nodes. It will be necessary to restore from a backup, which could mean that any changes since the last backup could be lost, and in some rare cases corruption can remain hidden for quite a while before it is discovered, and therefore even the backups could have this corruption.
  
  
• Directory Server Unable to Meet Load Performance Demands. If the Directory Server is unable to meet the performance demands of the clients, then this can be addressed in a replicated environment by adding additional replicas to handle the load. At present, this is really only an option for read operations, since write performance does not increase as additional replicas are added, but Directory Proxy Server 6 will also include a solution for providing write scalability as well. In a clustered environment, since only one node can be online at any time the only way to improve performance would be to buy bigger and/or faster hardware.
  
  
• Administrative Downtime. If a Directory Server instance needs to be made unavailable to perform some administrative action, then it will not have a significant impact in a replicated environment because all other instances can remain online to handle the requests. In a clustered environment, it depends on the nature of the administrative action to be taken. If the action is simply to restart the Directory Server then that can be done on the same machine without the need to fail over. If the system itself needs to be rebooted, then the service can be manually migrated to avoid the need to detect the failure which will add to the downtime. For potentially long-running operations that impact the Directory Server instance itself (e.g., restoring a backup, importing from LDIF, or rebuilding indexes), there is no choice but to keep the instance offline until that operation is complete.

Thursday May 18, 2006

A Quick Introduction to ASN.1 BER

• 00 -- This is the universal class. All of the "standard" BER elements have a universal type, so any time you see an element with a universal type you know what kind of data it holds. Examples of universal types include 0x01 (BOOLEAN), 0x02 (INTEGER), 0x04 (OCTET STRING), 0x05 (NULL), 0x0A (ENUMERATED), 0x30 (SEQUENCE), and 0x31 (SET). You'll notice that the binary encodings for all of those type values have the leftmost two bits set to zero.
  
  
• 01 -- This is the application-specific class. This class is used to allow an "application" to define its own types that will be consistent throughout that application. In this context, LDAP is considered an application. For example, any time you see 0x42 in LDAP, you know that it indicates an unbind request protocol op because RFC 2251 section 4.3 states that the bind request protocol op has a type of "[APPLICATION 2]".
  
  
• 10 -- This is the context-specific class. This class is used to indicate that the type is specific to a particular usage within a given application. The same type may be re-used in different contexts in the same application as long as there is enough other information for you to determine which context is applicable in a given situation. For example, in the context of the credentials in a bind request protocol op, the context-specific type 0x80 is used to hold the bind password, but in the context of an extended operation it would be used to hold the request OID.
  
  
• 11 -- This is the private class. I'm not aware of any cases in which it is used in LDAP.

• NULL -- The NULL element never has a value, and therefore the length is always zero.
  
  
• OCTET STRING -- The value of this element is simply encoded as a concatenation of the raw bytes of the data being represented. For example, to represent the string "Hello", the encoded value would be "48 65 6C 6C 6F". The value can have a length of zero bytes.
  
  
• BOOLEAN -- The value of this element is always a single byte. If all the bits in that byte are set to zero (i.e., 0x00), then the value is "FALSE". If one or more of the bytes is set to one, then the value is "TRUE". This means that there are 255 different ways to encode a BOOLEAN value of "TRUE", although in practice it's generally encoded as 0xFF (that is, all the bits are set to one).
  
  
• INTEGER -- The value of this element is encoded as a binary integer in two's complement form (see this article if you're not familiar with this representation). Although BER itself does not place a limit on the magnitude of the values that can be encoded, many software implementations have a cap of four or eight bytes (i.e., 32-bit or 64-bit integer values), and LDAP generally uses a maximum of 4 bytes (allowing you to encode values within the plus or minus 2 billion range). There will always be at least one byte in the value.
  
  
• ENUMERATED -- The value of this element is encoded in exactly the same way as the value of an INTEGER element.
  
  
• SEQUENCE -- The value of this element is simply a concatenation of the encoded BER elements contained in the sequence. For example, if I wanted to encode a sequence with two octet string elements encoding the text "Hello" and "there", then the encoded sequence value would be "04 05 48 65 6C 6C 6F 04 05 74 68 65 72 65". A sequence value can be zero bytes if there are no elements in the sequence.
  
  
• SET -- The value of this element is encoded in exactly the same way as the value of a SEQUENCE element.

04 05 48 65 6C 6C 6F
04 05 74 68 65 72 65

85 01 03

BindRequest ::= [APPLICATION 0] SEQUENCE {
     version                    INTEGER (1 .. 127),
     name                       OCTET STRING,
     authentication             CHOICE {
          simple                [0] OCTET STRING,
          sasl                  [3] SEQUENCE {
               mechanism        OCTET STRING,
               credentials      OCTET STRING OPTIONAL } } }

60 16 02 01 03 04 07 63 6E 3D 74 65 73 74 80 08 70 61 73 73 77 6F 72 64

• The first byte is 0x60 and it is the BER type for the bind request protocol op. It comes from the "[APPLICATION 0] SEQUENCE" portion of the definition. Because it's application-specific, then the class bytes will be 01, and because it's a SEQUENCE, then it will be constructed. If you put that together with a type value of zero, then the binary representation is "01100000", which is 0x60 hex.
  
  
• The second byte is 0x16, which indicates the length of the bind request sequence. 0x16 hex is 22 decimal, and if you count the number of bytes after the 0x16 then you'll see that there are 22 of them.
  
  
• Next comes "02 01 03", which is a universal INTEGER value of 3. It corresponds to the "version" component of the bind request sequence, and it indicates that this is an LDAPv3 bind request.
  
  
• Next comes "04 07 63 6E 3D 74 65 73 74", which is a universal OCTET STRING containing the text "cn=test". It corresponds to the "name" component of the bind request sequence.
  
  
• The last component is "80 08 70 61 73 73 77 6F 72 64", which is an element with a type of "context-specific primitive 0" and a length of eight bytes. From the definition of the bind request protocol op above, we can see that "context-specific" maps to the simple authentication type and that it should be treated as an OCTET STRING, and upon closer examination we can see that those eight bytes in the value do represent the encoded string "password".

Monday May 08, 2006

Tips for Developing LDAP Applications

• Make sure to use LDAPv3 rather than LDAPv2. Some APIs still default to LDAPv2, but LDAPv2 doesn't support features like controls, extended operations, referrals, SASL authentication, and multiple binds on the same connection.
  
  
• Use at least minimal caching to avoid repeating the same queries. If you include a list of attributes to return, then make sure that you include all attributes you may need rather than performing different queries to retrieve the same entry with different attribute lists.
  
  
• Design your application to allow for loose consistency in replication and the possibility that reads and writes may happen on different systems without the application's knowledge. Avoid read-after-write behavior because it can have inconsistent results.
  
  
• Don't treat the Directory Server like a relational database. Avoid splitting data into separate pieces so that you need to retrieve multiple entries to get all the information about a given entity.
  
  
• If you generate search filters, then do so intelligently. If you have compound filters, then use a form like "(&(a=b)(c=d)(e=f))" rather than "(&(&(&(a=b))(c=d))(e=f))" to avoid unnecessary nesting.
  
  
• Unbind connections when they're no longer needed. It's generally best to re-use connections as much as possible, but whenever you're done with a connection make sure it gets closed.
  
  
• Know the standards. There are a lot of them, but RFCs 2251 through 2256 will give a good overview. The specifications define what's legal and what isn't -- just because something happens to work doesn't mean that you should depend on it if it isn't a standard behavior.
  
  
• Learn to capture and interpret LDAP communication over the network. If your application is misbehaving then it will help dramatically if you can see the actual requests that it's sending and the responses that it is receiving. The LDAPDecoder tool provided with SLAMD has been designed specifically for this task, although other utilities like Ethereal or even Solaris snoop may be sufficient.
  
  
• Be directory-independent. Even though we'd prefer that you use Sun's directory, it's always best to design applications that are based completely on standard behavior. Even Sun's directory may change the way that certain features are implemented between releases, and as such you should be wary of proprietary features. If you do use features that are implementation-specific, then try to compartmentalize them into pluggable code that can be easily replaced if necessary.
  
  
• Use controls and extended operations wisely. Realize that not all servers support the same sets of extensions, and take that into account when designing the application. You can use the root DSE to determine whether the server supports a given control or extended operation. In some cases, it may be possible to provide a client-side implementation (e.g., client-side sort instead of server-side sort) as an alternative.
  
  
• Don't litter your code with hard-coded attribute/objectclass names, base DNs, server addresses/ports, usernames/passwords, etc. If you need to change something later, it can be hard to make sure that everything gets updated properly. You should centralize all such values in a constants class or a properties file so that they are simple to change if necessary.
  
  
• Where possible, maintain a set of persistent connections to the server (i.e., connection pools) rather than connecting and disconnecting for each operation. This will be much more efficient, especially when using SSL. In order to avoid leaking connections and duplicating large amounts of code, it may be a good idea to code the various types of operations into the connection pool itself so that those operations will check out a connection, perform the operation and any necessary error handling, and make sure the connection is put back into the pool.
  
  
• If you do use pooled connections, then use the proxied authorization control (if the server supports it) to avoid the need to constantly rebind as a given user in order to perform operations.
  
  
• Design your application to be able to handle the different kinds of failures that may arise: server down, network outage, DS backlogged or unresponsive, DS returning unexpected responses (e.g., unavailable or busy). Don't assume that a lost connection means the server is down -- it could be that the connection was closed due to the idle timeout or some other constraint.
  
  
• In some environments, it may be necessary to allow for the possibility of failing over to a read-only server. In this case, your application should be able to follow referrals, and also to handle cases where none of the referred servers are available. Also, consider adding a read-only mode that could allow your application to continue with at least partial functionality in the even that no writable servers are available.
  
  
• Consider the authentication mechanisms you might need to support. Simple authentication (bind DN and password) is virtually guaranteed to be supported, but may require SSL/StartTLS. In some cases (depending on access control configuration), SASL authentication may be required for some operations. You should never try to retrieve the hashed password and compare it against what the user provided because this can introduce significant security holes in your application and can bypass password policy and account lockout constraints.
  
  
• When binding, make sure that the user actually specified a password, since simple binds that don't contain a password will be treated as anonymous. Consider using the authorization identity controls (RFC 3829) or the LDAP "Who Am I?" extended operation (draft-zeilenga-ldap-authzid) to ensure that the authentication was actually performed as the appropriate user and isn't anonymous.
  
  
• When binding, make sure to check for password policy controls to see if the password has expired or will expire in the near future. Don't design your application to expect hard-coded result codes or error messages to figure out the reason for the bind failure.
  
  
• Work within the access control constaints of the underlying Directory Server. Don't perform all operations as an administrator, as that may open security holes and can make auditing difficult or impossible. Avoid programmatic interaction with the ACIs of the underlying server because they are non-standard and the syntaxes may vary between servers or even between releases of the same server. Keep access controls simple and avoid using too many of them to reduce performance impact and preserve clarity. You may be able to use the GetEffectiveRights control to determine what rights the client has.
  
  
• When designing your application, make sure to document the kinds of operations that may be performed, both individually and in sequences of operations. Consider developing a SLAMD job that can simulate the access patterns of your application to help administrators better understand the impact that changes to the server configuration might have on the applications using it.
  
  
• Talk with the server administrators to discuss any schema or indexing changes that your application may require. Let them know about any controls or extended operations you might want to use to ensure they are supported and that they won't significantly hurt server performance. Also indicate anticipated usage levels to ensure that the administrators can prepare for the increased load that the application may cause.
  
  
• Use groups effectively. Prefer dynamic groups over static groups since they are much easier to maintain and faster to work with. Avoid roles altogether, since they are non-standard functionality and don't really provide much benefit. If you must use static groups, then let the server determine the membership with a filter like "(|(member=userdn)(uniqueMember=userDN))" instead of retrieving the entire member list and trying to make the determination on the client side.
  
  

Monday May 01, 2006

Breaking Up (Directory Data) is Hard to Do

• As mentioned above, introducing new hierarchy is almost always a bad idea. It can break client applications that don't deal well with the additional branching, particularly those that use onelevel searches or try to add new entries or otherwise construct DNs. It also creates a maintenance problem for cases in which the split is made based on criteria that can change (e.g., if the data is branched based on geographic location and one of the users moves to a different region).
  
  
• Certain types of operations don't always work as expected if it is necessary to cross database boundaries. For example, the server-side sort operation (and anything that depends on it, like the virtual list view) would cause the entries to be sorted within each database but not between the databases.
  
  
• Separating the data into multiple databases in the same instance will help reduce the length of time required to perform an LDIF import or export, as well as related operations like rebuilding indexes. However, it won't do anything to impact the length of time required to back up and restore the data because all databases (including the transaction logs) must be archived and restored together.
  
  
• In most cases, using multiple databases will not do anything to help improve the performance of read operations. The primary case in which read performance might benefit would one in which the database is larger than will fit in memory and therefore most reads will need to go to disk. In this case it's likely that the server will be I/O bound, and putting each database on a separate disk subsystem will help increase the total number of operations that can be performed before the I/O saturation point.
  
  
• Using multiple databases does not really help write performance when viewed from the perspective of disk I/O. In most cases, the only time that the actual database files will be written is during a checkpoint (the rest of the time, the updates go to the transaction logs), and all the writes are ordered by database file and written serially so none of the database files are updated in parallel. Even if the databases were separated onto different disk subsystems, they would still be written one at a time so when one was busy the others would be largely idle.
  
  
• Using multiple databases actually can help write performance to an extent because there is a significant amount of lock contention in write operations that happens at the backend level. If two writes are targeted at two different databases then there will be a lot less contention than if they had been in the same database and therefore more of the updates will be able to be performed in parallel. The amount of lock contention was reduced significantly in the 5.2 patch 4 release as compared with earlier versions, and it should be reduced even further in the upcoming 6.0 release, so the benefits to write performance from splitting into multiple databases will be diminishing in the future.