Thursday May 25, 2006
Thursday May 25, 2006 Replication versus Cluster for DS High Availability
In many environments, the directory service is mission critical. If it's down or not working properly, then it will cause all kinds of problems. It can prevent your customers from logging into their online accounts. It can prevent employees from getting their e-mail or logging into servers and workstations. In some environments, it could even mean that parts won't show up at the factory on time or that the door access control systems won't work. In almost all environments, a directory service outage can mean lost revenue and/or productivity, so it is vital that the service stays up all the time.There are a lot of aspects to ensuring a highly-available directory service and I won't attempt to cover all of them here. But one topic that keeps coming up from time to time is the question of whether to use the Directory Server replication features or to use a high availability cluster or some other external HA solution like storage-level replication. In this post, I'll detail why Directory Server replication is clearly the better choice. It will be presented primarily in the context of Directory Server versus an HA cluster, but many of the arguments may apply to alternate solutions as well. I will also not address the possibility of using replication in which each instance is a part of its own cluster, as that can be a very expensive solution and doesn't really offer much benefit outside of just usting replication by itself.
First, let's take a quick look at Directory Server replication. It is the means by which changes in the data are kept in sync across multiple instances. Whenever a change is made on a master Directory Server instance, it will be propagated to all other servers in the environment. In the current 5.2 patch 4 release, you can have up to four masters capable of accepting changes and any number of read-only replicas that will automatically refer any write attempts to the masters. In the upcoming Directory Server 6.0 release, it will be possible to have any number of masters. Each replica (whether read-only or read-write) has its own copy of the database, and can have its own settings for things like indexing and cache sizing. We have a sophisticated conflict detection and resolution mechanism that helps ensure that concurrent changes to the same entry on different servers are handled properly. The total directory service load can be split across all of the instances, and if you're using Directory Proxy Server (which is included as part of DSEE at no additional cost) if one of the Directory Server instances becomes unavailable for some reason client requests can be transparently redirected to other servers. Replication is a core part of the Directory Server product (i.e., not an add-on that requires additional fees or licensing) that is fully supported and is in heavy use by virtually all of our customers.
On the other hand, HA cluster products attempt to achieve high availability through monitoring and failover. The service is running on one system at any given time, but if a problem occurs that would make it unavailable (e.g., a hardware failure or OS crash) then it will be migrated to another machine in the cluster. When it fails over to another system, it will retain the same network settings like IP address and hostname so there isn't need for an external load balancer or Directory Proxy Server, and it will use the same storage so there isn't a need to have multiple copies of the data. However, the Directory Server instance can only be running on one node of the cluster at any given time, and there will be an interruption in service for the time required to detect the failure and migrate the Directory Server instance.
So already, we've seen a few differences that make Directory Server replication much more attractive than clustering. In the event of a failure, a "high availability" cluster actually guarantees downtime because it does take time for it to detect the failure and migrate to another node. Since there's a single instance running in the cluster, you can't take advantage of all the hardware concurrently to get improved performance. And since it's using the same storage then you can't have different indexing or other database-level configuration differences.
To examine the issue further, let's look into the different failure modes that can impact a directory service:
-
Server Hardware Failure. In the event that a hardware failure leaves a system offline or in a degraded state, clients in a replicated environment could have instantaneous and transparent failover to another instance. A clustered environment can handle this failure as well, since the service would be migrated to another node, although there would be some downtime.
-
Operating System Crash or Failure. With either replication or clustering, an OS crash is handled in essentially the same way as a hardware failure. If the OS hangs rather than crashes, then clients in a replicated environment would need to have some kind of timeout, after which the request can be redirected to another replica (the Directory Proxy Server can handle this transparently). In a clustered environment, an OS hang would be essentially treated in the same way as an OS crash.
-
Storage Hardware Failure. If the storage hardware failure is catastrophic and results in the loss of all access to the data, then a replicated environment would be able to survive because each Directory Server instance has its own copy of the database on its own storage. In a clustered environment, it would not be possible to handle this because the nodes share the same storage. It is true that a failure like this is pretty unlikely if the storage itself has been designed with a highly-available configuration using RAID and redundant devices, but in that case a failure of one component could lead to degraded performance for some period of time (e.g., if RAID 5 needs to reconstruct data from parity for all reads, and there would be an I/O performance hit when rebuilding the failed component after it has been replaced).
-
Whole Site Failure / Unavailability. If an entire site becomes unavailable for some reason (e.g., catastrophic network failure, extended power outage, natural disaster, etc.), then clients in a replicated environment can access other servers in other sites because replication works well in WAN environments. Clustering is a lot more complicated in WAN environments, and if it is even feasible in this kind of outage (e.g., when combined with replicated storage) it may result in a scenario in which multiple nodes are providing the service concurrently and there can be irreconcilable conflicts if multiple nodes allow data to be updated.
-
Directory Server Crash. If the Directory Server process crashes, then clients in a replicated environment can be transparently redirected to other instances that are still online. In a clustered environment, there will be an outage until the Directory Server is restarted. In most cases it will be faster to restart the server on the same node than to migrate it to a different node, so the cluster doesn't provide much benefit there other than detecting the failure.
-
Directory Server Hang / Freeze. If the Directory Server process hangs, then clients in a replicated environment can be transparently redirected to other instances that are still functional after some timeout is detected. Since the service itself is still available, the hung instance can be left in that state to see if it recovers itself (in most cases, this is simply a case in which all of the worker threads have been tied up processing expensive requests and the server will become responsive once processing on those requests has completed) or to capture diagnostic information like stack traces and/or core files that could help understand the cause of the hang. In a clustered environment, since there is only one instance it must be killed and restarted immediately to minimize the service outage, which prevents the opportunity for the server to recover itself, and which can inhibit the ability to identify the cause of the hang. In many cases, killing and restarting on the same system will be faster than killing and migrating to a different node. If the service does need to be forcefully killed, then it will also take additional time to restart as it may need to verify the integrity of the database.
-
Directory Server Database Corruption. If the Directory Server database becomes corrupt for some reason in a replicated environment, then it will only impact one instance since all instances have their own database and replication is based on sending LDAP changes, not replicating at the database level. A clustered environment is not able to handle this kind of failure because it uses the same storage and database instance when failing over between nodes. It will be necessary to restore from a backup, which could mean that any changes since the last backup could be lost, and in some rare cases corruption can remain hidden for quite a while before it is discovered, and therefore even the backups could have this corruption.
-
Directory Server Unable to Meet Load Performance Demands. If the Directory Server is unable to meet the performance demands of the clients, then this can be addressed in a replicated environment by adding additional replicas to handle the load. At present, this is really only an option for read operations, since write performance does not increase as additional replicas are added, but Directory Proxy Server 6 will also include a solution for providing write scalability as well. In a clustered environment, since only one node can be online at any time the only way to improve performance would be to buy bigger and/or faster hardware.
- Administrative Downtime. If a Directory Server instance needs to be made unavailable to perform some administrative action, then it will not have a significant impact in a replicated environment because all other instances can remain online to handle the requests. In a clustered environment, it depends on the nature of the administrative action to be taken. If the action is simply to restart the Directory Server then that can be done on the same machine without the need to fail over. If the system itself needs to be rebooted, then the service can be manually migrated to avoid the need to detect the failure which will add to the downtime. For potentially long-running operations that impact the Directory Server instance itself (e.g., restoring a backup, importing from LDIF, or rebuilding indexes), there is no choice but to keep the instance offline until that operation is complete.
I could go on, but hopefully I've made my point. High availability clusters are very good in many environments, but they are not as well-suited for keeping your directory service up and running as Directory Server replication. In addition, we are continuing to make improvements in the area of replication to make it easier to use and monitor, to offer better performance and lower latency, and to include more features, so the advantages of using replication will only continue to grow.
Posted by cn_equals_directory_manager ( May 25 2006, 10:26:20 AM CDT ) Permalink Comments [2]
Post a Comment:
Comments are closed for this entry.
| Archives | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Language | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Links | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Referrers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Today's Page Hits: 89 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Another solution for us was to use a hardware network load balancer (Foundry ServerIron XL). While this will work in some cases, we experienced some inconsistent behavior and had no other option than to go the HA route.
So, having said all that, how would you go about dealing with an application that can only point to a single instance of directory server?
Posted by Jeff on May 25, 2006 at 02:38 PM CDT #
As I mentioned above, Directory Proxy Server is included as a part of DSEE, so there's no extra cost, and if you're not maxing out the Directory Server hardware you can even run an instance of DPS on each of the Directory Server systems with a hardware load-balancer in front of them.
If you have applications that don't work properly through the Directory Proxy Server, then we'd like to know about them so we can see what needs to be done to support them.
Posted by Neil Wilson on May 25, 2006 at 02:55 PM CDT #