Trond Norbye's Weblog

Connection pooling libmemcached

A while back I looked at the Memcached UDF for MySQL, and noticed that it didn't use libmemcached in an optimal way. In order to work in a multithreaded environment it used the following pattern:

   memcached_st* clone = memcached_clone(NULL, memc);

   ... memcached operations using the clone ---   

   memcached_free(clone);

Well, that doesn't look bad, does it? Well, it isn't that bad, but if you look at the network traffic you will see that we end up connecting / disconnecting to the involved memcached servers every time, and memcached is not optimized for "single-shot" connections.

So how should you solve this? Well, you should reuse your clones! And luckily for you, you don't have to reinvent the wheel. Yesterday I pushed a patch to libmemcached introducing a new library: libmemcachedutil. The intention of that library is to put utility functions built on top of libmemcached that you might want to use in your application, and the first routine there is the pool functionality.

So let's write some code using the new library:

#include <pthread.h>
#include <stdbool.h>
#include <signal.h>

#include "libmemcached/memcached_util.h"


static volatile bool run = true;

static void sig_handler(int sig) {
    assert(sig == SIGINT);
    run = false;
}

static void* my_application_thread(void *arg) 
{
  memcached_pool_st* pool = arg;

  while (run) {
    memcached_return rc;
    memcached_st* mem = memcached_pool_pop(pool, true, &rc);

    if (mem != NULL) {
      ... use the memcached handle for whatever you want! ...

      /* Return the instance to the pool */
      if (memcached_pool_push(pool, mem) != MEMCACHED_SUCCESS) {
        fprintf(stderr, "Failed to release the memcached instance!\n");
      }
    } else {
      fprintf(stderr, "Failed to get the memcached instance from pool!\n");
    }
  }

  return NULL;
}

int main(int argc, char** argv)
{
  memcached_st* memc = memcached_create(NULL);
  if (memc == NULL) {
    fprintf(stderr, "Failed to create memcached instance\n");
    return 1;
  }

  if (memcached_server_add(memc, "localhost", 11211) != MEMCACHED_SUCCESS) {
    fprintf(stderr, "Failed to add localhost to the server pool\n");
    memcached_free(memc);
    return 1;
  }
 
  memcached_pool_st* pool= memcached_pool_create(memc, 5, 10);
  if (pool == NULL) {
    fprintf(stderr, "Failed to create connection pool\n");
    memcached_free(memc);
    return 1;
  }

  signal(SIGINT, sig_handler);

  /* create 10 threads to use the pool */
  pthread tid[10];
  for (int x= 0; x < 10; ++x) {
    pthread_create(&tid[x], NULL, my_application_thread, pool);
  }

  for (int x= 0; x < 10; ++x) {
    pthread_join(&tid[x], NULL);
  }

  /* Release allocated resources */
  memcached_pool_destroy(pool);
  memcached_free(memc);

  return 0;
}

Posted at 10:26AM May 12, 2009 by trond in Memcached  |  Comments[0]