/var/adm/blog

basic UDP support in libmemcached

Yesterday I submitted a patch to libmemcached, the popular Memcached client library. The patch provides basic, yet limited, support for using the User Data Protocol (UDP) when communicating with a Memcached server. For those of you interested, you can get the source and documentation at http://hg.tangent.org/libmemcached .

In this post, I will describe how this patch behaves and describe some of the rational for how it was implemented and how it should be used.

UDP SUPPORT IN MEMCACHED SERVER

The Memcached Protocol Specification (http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt), does allow for UDP communication with the Memcached server. The Memcached UDP specification is a super-set of the TCP specification, it follows the same format as the Memcached server TCP protocol with the following additions:

1. each request must be entirely contained within a single UDP data-gram.
2. each data-gram request must include an 8 byte frame header (see specification for details). Components of this header provide basic session management as well as a means to ensure that the client is not violating the Memcached server UDP protocol, for example by sending multi-data-gram requests.
3. When responding to a request, the server is not limited to a single data-gram. The response may be a multi-datagram request.

UDP SUPPORT IN LIBMEMCACHED

Limited Operations

Each of the constraints placed upon UDP communications with the Memcached server (see above), combined with the stateless/connectionless nature of the UDP protocol, ended up having an impact on how UDP support was implemented in libmemcached.  

The most significant impact was the decision to limit the type of operations which could be executed when running in UDP mode.  There exists a set of operations in libmemcached that issue a request of the server and then require a response from the server, for example memcached_get(). The fact that the nature of these types of operations require a response from the server posses a difficulty since UDP gives us no assurance that either the request or the response will be delivered. That the server may respond to a request with a multi-datagram message further complicates the matter since UDP does not ensure proper ordering of data-grams that have been received.  Since ordering and delivery assurance is the purview of TCP and not wanting to rewrite TCP with in libmemcached, we decided to simply not support these types of operations in UDP mode. As such the following functions are not available when using UDP communication:

memcached_version(), memcached_stat(), memcached_get(), memcached_get_by_key(), memcached_mget(), mem-cached_mget_by_key(), memcached_fetch(), memcached_fetch_result(), memcached_value_fetch().  All other operations are supported.

Fire-and-Forget

As is hinted to in the section above, the UDP implementation does not attempt to handle responses from the server, the rationale being that since we can not be assured to receive the response, we should assume that we wont receive it. Because of this assumption, all supported operations are executed in a "fire-and-forget" mode, by which it means that once the operation has been executed by the client, no attempt is made to ensure that the operation was received or executed by the server. Since no attempt will be made to handle the server's response, when executing an operation in UDP mode the 'noreply' option is sent for all operations which support noreply. 

Limited Size 

In UDP mode, the current implementation limits the size of cache entries that can be sent to the server to 1KB. Technically, the limit of user supplied data is above 1300 bytes, the exact limit depending on whether the binary or ASCII protocol is being used and if the ASCII protocol is being used then the limit depends on which command is being executed (set, replace, cas, etc.). For simplicity's sake the user should consider the limit to be 1KB, although the client will allow operations above 1KB as long as the entire message --include UDP header and command overhead, does not exceed 1400 bytes.  This limit is a consequence of the fact that the server does not support multi-datagram requests, which implies that no cache entry may be larger than a data-gram. Currently, the server defines the maximum data-gram size as 1400 bytes and to be consistent with the server, our implementation abides by this limit (it should be noted that the server currently only uses this limit when crafting UDP responses --which we ignore-- so it is conceivable that we could raise the client-side size limit to a larger value).

No Mixing

The primary data-structure in the libmemcached library is memcached_st, which acts a handle for the client. The typical use pattern is to create an 'instance' of this structure (by calling memcached_create() ) and then adding server 'instances' to this structure for each Mecached server we wish to communicate with  (via memcached_add_servers()). Our UDP implementation does not allow  for UDP and TCP servers to be added to the same memcached_st client instance. When we consider the fact that UDP server instances will not be able to support all operations (such as get, see above) we can see how mixing UDP and TCP servers in the same client would be problematic. As an example, let us assume we have added ServerA and ServerB to the same client instance. If ServerA were using UDP while serverB used TCP, than any get operation who's key hashed to ServerA would fail, since UDP get is not supported, while those that hash to Server B would pass (assuming the key exists in the cache).  This inconsistent behavior is undesirable and as such the mixing of UDP and non-UDP servers in the same client 'instance' is not allowed. Should an application need to use both UDP and TCP to communicate with its Memcached servers, this can be accomplished by using two separate memcached_st client 'instances', one for each transport protocol (see examples below). 

EXAMPLES 

The example below shows how to set up a libmemcached client which communicates with the server via UDP 

  

  memcached_st *memc;

  //creates handle for client instance

  memc= memcached_create(NULL);

  //turns on udp behavior for the memc client handle

  memcached_behavior_set(memc, MEMCACHED_BEHAVIOR_USE_UDP, 1);

  //add the server 127.0.0.1:11211 to the client

  memcached_server_add_udp(memc, "127.0.0.1", 11211); 

  //store some data

  char *key= "foo";

  char *value= "when we sanitize";

  memcached_return rc= memcached_set(memc, key, strlen(key),value, strlen(value),(time_t)0, (uint32_t)0);

This example shows how to set up two clients, one using UDP and one using TCP

  memcached_st *udp_client;

  memcached_st *tcp_client;

  //creates udp handle for client instance

  udp_client= memcached_create(NULL);

  //creates tcp handle for client instance

  tcp_client= memcached_create(NULL);

  //turns on udp behavior for the udp client handle

  memcached_behavior_set(udp_client, MEMCACHED_BEHAVIOR_USE_UDP, 1);

  //add the server 127.0.0.1:11211 to the clients

  memcached_server_add_udp(udp_client, "127.0.0.1", 11211); 

  memcached_server_add(tcp_client, "127.0.0.1", 11211); 

  //store some data

  char *key= "foo";

  char *value= "when we sanitize";

  memcached_return rc= memcached_set(udp_client, key, strlen(key),value, strlen(value),(time_t)0, (uint32_t)0);

  //get some data; note we use the tcp client

  size_t vlen;

  uint32_t flags;

  char * value= memcached_get(tcp_client, key, strlen(key),&vlen,&flags,&rc);

FUTURE

We recognize this is a fairly basic implementation of UDP support in libmemcached; the idea was to start small and use feedback from this implementation to determine what a more fully formed implementation should look like or if one is even needed at all. So if you are interested, please download the bits, kick the tires and tell us what ya think.

Posted at 09:14PM Mar 10, 2009 by Eric Lambert in Memcached  |  Comments[0]