Trond Norbye's Weblog

Replicate your keys to multiple memcached servers.

If you look at a how the (community version) of memcached works, all servers are completely isolated from each other. They don't know (or care) about the existence of other servers, and all advanced logic is implemented by the clients. This removes a lot of complexity from the server, resulting in a small clean source base with few bugs. You will also find this simple design in the client-server protocol, reducing what you can try to implement in the server.

If you scan the mailing lists you will find that requests for replication seems to pop up with a regular interval, so I decided to give it a shot. Personally I am not too interested in a full replicated scenario (where you have all of your keys stored on multiple machines), because I think you would be wasting too much space. I think a mixed mode is more interesting, where you store only a few of the items on multiple servers; and this is what I implemented.

If you look at the design for the replication from a 1000ft, it is dead simple. When we store a key on the server, we will also store it on the n'th next servers. If we encounter a problem when we try to send the GET request to the server we try fetch the replica instead. We will however not try to fetch the replica if:

• The server crash before sending the response back. This will result in a cache miss (because we don't have any state withing libmemcached to recreate GET-request so that we can send it to the next replica server.)
• The server doesn't have the item. A cache miss will be returned to the caller immediately, because trying the replica servers would cause long delays for real cache misses.

If you want to try it out you need to grab at least revision 539, but you should be aware of some design choices / limitations:

• It is only supported with the binary protocol, so you cannot use a memcached server from the 1.2 series (you need the 1.4 branch).

  Why? Well the replication code use the "noreply" mode to store the replicas, and the "noreply" mode in the ASCII protocol is just one big hack ;-)

• SET is the only command that will store multiple replicas.

  The replication code does not implement any kind of transactions / consistency, so I wanted to expose this fact to the user. Allowing ADD or REPLACE could confuse the users and introduce strange bugs in their application. INCR and DECR raise the same inconsistency problems. If you have an atomic counter (at least if it doesn't get evicted from the cache) you don't want it to behave strangely because of race conditions updating the replicas.

• The CAS identified is generated on the server, so the master item and all replicas will have different CAS identifiers. If you enable replication you can't use CAS
• We don't detect (and handle) network partition
• If you run several memcached instances on the same server, you don't want to list them next to each other in the server list. The replication works in such a way that it will hash the key to locate the server the object belongs on, and it will store it on the n next servers in the list. If you list memcached instances on the same server next to each other, you might end up having the master and all of the replicas on the same server.
• If you use consistent hashing you can grow your pool without blowing the complete cache
• The replication works on a per memcached_st instance, so the API stays the same (and adds no extra costs if you don't use it

Well, I guess a lot of you don't like reading text that don't end each statement with a semicolon, so I should probably add some code. First you should locate the code where you create your memcached_st handle. You probably have something like (I removed the error checking to keep the example small, but you don't want to do that in your code!!!!):

   memcached_st *memc = memcached_create(NULL);
   memcached_server_st *servers = memcached_servers_parse(server_list);
   memcached_server_push(memc, servers);
   memcached_server_list_free(servers);

The first thing we need to do is to enable the binary protocol:

   memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_BINARY_PROTOCOL, 1);

As I mentioned above, I don't think you really want to replicate all of your keys, so let's create a new memcached_st instance and enable replication there (num_replicas contains the number of replicas I want):

   memcached_st repl = memcached_clone(NULL, memc);
   memcached_behavior_set(repl, MEMCACHED_BEHAVIOR_NUMBER_OF_REPLICAS, num_replicas);

And that's all you need to do! If you want to store a key with multiple replicas, you would go ahead and store it using the repl instance. For "normal" items, you would use the memc instance:

  /* Store a key with replicas: */
  memcached_set(repl, "replicated", 10, "foo", 3, 0, 0);
  /* Try to get the item (or the replicas if we have problems talking to the master) */
  void* value = memcached_get(repl, "replicated", 10, &vlen, &flags, &rc);
  /* Store a without replicas */
  memcached_set(memc, "single", 6, "foo", 3, 0, 0);
  /* Try to get the item */
  void* value = memcached_get(memc, "single", 6, &vlen, &flags, &rc);
  /* We can also get the master of a replicated item: */
  void* value = memcached_get(memc, "replicated", 10, &vlen, &flags, &rc);

Posted at 11:50AM Jun 25, 2009 by trond in Memcached  |  Comments[1]