Bazaar shared repositories and Hudson build slaves

I created a Bazaar plugin for Hudson a while back, and we have been using that with great success on our Hudson build farm to build Drizzle, libmemcached and Gearman.

If you look at our build farm you will see that we compile the same source project multiple times on the same slaves, only with a variation in the configuration (compiler, 32/64 bit etc). With the normal configuration you will check out the complete repository for each of the projects, using a lot of bandwidth and disk space. Disk is cheap these days so I don't care that much about that, but my bandwidth is limited so I would like to reduce that if possible.

If we look at the disk layout on the build slaves it perfectly suited for using Bazaar shared repositories:

$SLAVE_HOME/workspace/

So if we create a Bazaar shared repository in $SLAVE_HOME/workspace, all of our Bazaar projects will store the upstream information in $SLAVE_HOME/workspace/.bzr. Not only will this dramatically reduce the disk footprint, it will also reduce the amount of data being downloaded. So how do you set up a shared repository? It's no magic, simply log into the build slave and run the following command:

trond@storm > bzr init-repository $SLAVE_HOME/workspace

Unfortunately there is a "bug" in Hudson causing Hudson to nuke the directory every day when it is doing housekeeping of the workspace directory. Luckily for us there is a simple workaround for the issue, simply create a project named ".bzr".

Update: Hudson will also nuke the directory if it hasn't been modified on a time period, and it seems that bazaar doesn't update the modified date on the .bzr directory. A simple workaround for this is to install the following cron job:

1 1 * * 0 /usr/bin/touch /home/hudson/hudson/workspace/.bzr

Posted at 09:53AM Oct 17, 2009 by trond in OpenSolaris  |  Comments[0]

memcapable, part two

Today I added support for the ASCII protocol into memcapable so that it may be used to test both the binary and the ASCII protocol. By running it on the example server I added with the protocol parser in libmemcached I discovered that it failed all tests (mostly due to incorrect handling of noreply, but that is another story). It is not merged into trunk yet, so if you want to play with it today you need to branch lp:~trond-norbye/libmemcached/bugparade.

This is the result from running it on the server in my sandbox:

trond@storm> ./memcapable
ascii quit                              [pass]
ascii version                           [pass]
ascii verbosity                         [pass]
ascii set                               [pass]
ascii set noreply                       [pass]
ascii get                               [pass]
ascii gets                              [pass]
ascii mget                              [pass]
ascii flush                             [pass]
ascii flush noreply                     [pass]
ascii add                               [pass]
ascii add noreply                       [pass]
ascii replace                           [pass]
ascii replace noreply                   [pass]
ascii cas                               [pass]
ascii cas noreply                       [pass]
ascii delete                            [pass]
ascii delete noreply                    [pass]
ascii incr                              [pass]
ascii incr noreply                      [pass]
ascii decr                              [pass]
ascii decr noreply                      [pass]
ascii append                            [pass]
ascii append noreply                    [pass]
ascii prepend                           [pass]
ascii prepend noreply                   [pass]
ascii stat                              [pass]
binary noop                             [pass]
binary quit                             [pass]
binary quitq                            [pass]
binary set                              [pass]
binary setq                             [pass]
binary flush                            [pass]
binary flushq                           [pass]
binary add                              [pass]
binary addq                             [pass]
binary replace                          [pass]
binary replaceq                         [pass]
binary delete                           [pass]
binary deleteq                          [pass]
binary get                              [pass]
binary getq                             [pass]
binary getk                             [pass]
binary getkq                            [pass]
binary incr                             [pass]
binary incrq                            [pass]
binary decr                             [pass]
binary decrq                            [pass]
binary version                          [pass]
binary append                           [pass]
binary appendq                          [pass]
binary prepend                          [pass]
binary prependq                         [pass]
binary stat                             [pass]
binary illegal                          [pass]
All tests passed

I just discovered while reading the spec one more time today that some of the tests are not according to the spec, so I am going to submit bug reports on the community server and fix the tests:

• verbosity should always return OK, but does not if it encounter illegal number of options
• flush_all does not fail for illegal options
• delete a b does not fail, but assumes that the second option "noreply"

Do you see any other bugs in the ASCII protocol handling in the community server?

Posted at 03:40PM Oct 07, 2009 by trond in Memcached  |  Comments[0]

memcapable

Earlier today Matt Ingenthron blogged about the new tool memcapable I wrote a while back. In his blog Matt mentions some of the reasons why we want such a tool, but he didn't actually mention the reason for why I actually sat down to create the tool.

If you follow my blog you might remember my entry "Callback based protocol parser in libmemcached?". Before I could start implementing the parser, I really needed a tool to:

1. send all of the defined packet structures to the server
2. verify the response packet generated from the server

I didn't have the need for the textual protocol at the time I wrote the initial version of memcapable, so right now memcapable can only be used to test the binary protocol (not all variants of all commands are implemented in the initial version).

So how does it work? In it's simplest form you can use it to test the memcached server running on the default port on the same computer:

trond@storm> ./memcapable
noop		[pass]
quit		[pass]
quitq		[pass]
set		[pass]
setq		[pass]
flush		[pass]
flushq		[pass]
add		[pass]
addq		[pass]
replace		[pass]
replaceq		[pass]
delete		[pass]
deleteq		[pass]
get		[pass]
getq		[pass]
getk		[pass]
getkq		[pass]
incr		[pass]
incrq		[pass]
decr		[pass]
decrq		[pass]
version		[pass]
append		[pass]
appendq		[pass]
prepend		[pass]
prependq		[pass]
stat		[pass]
illegal		[pass]
All tests passed

Now this looks really nice doesn't it, but let's try to run it on a 1.2.8 version:

trond@storm> ./memcapable
noop		[FAIL]
quit		[FAIL]
quitq		[FAIL]
set		[FAIL]
setq		[FAIL]
flush		[FAIL]
flushq		[FAIL]
add		[FAIL]
addq		[FAIL]
replace		[FAIL]
replaceq		[FAIL]
delete		[FAIL]
deleteq		[FAIL]
get		[FAIL]
getq		[FAIL]
getk		[FAIL]
getkq		[FAIL]
incr		[FAIL]
incrq		[FAIL]
decr		[FAIL]
decrq		[FAIL]
version		[FAIL]
append		[FAIL]
appendq		[FAIL]
prepend		[FAIL]
prependq		[FAIL]
stat		[FAIL]
illegal		[FAIL]
28 of 28 tests failed

This shouldn't come as a big surprise, because the binary protocol isn't implemented in 1.2.8. Getting [FAIL] isn't really that informative, because it doesn't help you as a developer to figure out what's wrong. I have added a couple of options to the program that may help you to track down the real problem: -v and -c.

-v

Print out the assertion that failed

-c

Create a coredump when an assertion fails

Let's start the memcached server and disable the use of CAS and re-run memcapable with -v -c

trond@storm> ./memcapable -v -c
noop		[pass]
quit		[pass]
quitq		[pass]
set		memcapable.c:493: rsp->plain.message.header.response.cas != 0
zsh: IOT instruction (core dumped)  ./memcapable -v -c

As you can see it expects the response packet to have a CAS value set for the operation. If you would like to inspect the response packet you could load it into your debugger and poke around:

trond@storm> dbx - core
Corefile specified executable: "/source/libmemcached/memcapable/clients/memcapable"
Reading memcapable
core file header read successfully
Reading ld.so.1
Reading libm.so.2
Reading libnsl.so.1
Reading libsocket.so.1
Reading libpthread.so.1
Reading libthread.so.1
Reading libc.so.1
t@1 (l@1) program terminated by signal ABRT (Abort)
0xfffffd7fff2842aa: _lwp_kill+0x000a:	jae      _lwp_kill+0x18	[ 0xfffffd7fff2842b8, .+0xe ]
Current function is ensure
  212         abort();
(dbx) where
current thread: t@1
  [1] _lwp_kill(0x1, 0x6, 0xffffff01cdf92ae0, 0xfffffd7fff284c0e, 0x12, 0x0), at 0xfffffd7fff2842aa 
  [2] thr_kill(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff2788cd 
  [3] raise(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff227511 
  [4] abort(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff1fda41 
=>[5] ensure(val = 0, expression = 0x408498 "rsp->plain.message.header.response.cas != 0", file = 0x408198 "memcapable.c", line = 493), line 212 in "memcapable.c"
  [6] do_validate_response_header(rsp = 0xfffffd7fffdfeed8, cc = '\001', status = 0), line 493 in "memcapable.c"
  [7] test_binary_set_impl(key = 0x4087b0 "test_binary_set", cc = '\001'), line 621 in "memcapable.c"
  [8] test_binary_set(), line 651 in "memcapable.c"
  [9] main(argc = 3, argv = 0xfffffd7fffdff798), line 1196 in "memcapable.c"
(dbx) frame 6
Current function is do_validate_response_header
  493         verify(rsp->plain.message.header.response.cas != 0);
(dbx) print rsp->plain.message.header.response
rsp->plain.message.header.response = {
    magic    = '�'
    opcode   = '\001'
    keylen   = 0
    extlen   = '\0'
    datatype = '\0'
    status   = 0
    bodylen  = 0
    opaque   = 3735928559U
    cas      = 0
}

I noticed earlier today that there are some issues with the -v flag if you don't include -c (it just prints out the assertion and reports everything back up as success ;-)). I'll try to address that in the next release.

Posted at 11:39PM Sep 21, 2009 by trond in Memcached  |  Comments[0]

Trailing whitespace

If you are an emacs user and don't want your source code to include trailing whitespaces, try adding the following to your ~/.emacs file:

(add-hook 'before-save-hook 'delete-trailing-whitespace)

This will automatically remove all trailing whitespaces from your buffer before saving your file!

Posted at 10:30AM Sep 15, 2009 by trond in OpenSolaris  |  Comments[0]

Don't log, dump core!

A common trend as a profession grows and diversifies is the loss of the good, old craftsmanship; software development is no exception. It seems to me developers who use a debugger are a dying breed, and many who do more than "rm" on a corefile are really hard to find. So what's wrong with logging?? Well, I'll suggest that you start off by reading Don't Log, Debug!

I think Tor has some really good points in his blog post. When you write the program you don't know where the bug is, so you will most likely not include enough information to track down the bug anyway. You will most likely have to provide the customer with an instrumented version if you cannot reproduce the problem locally.

I have heard people trying to excuse themselves by saying: "I can't use a debugger to find this issue because it is a timing issue". Well, I don't buy that, because if they have to add more logging from their code the timing will change as well (and possibly mask out the error).

Tor works in "the Java world", whereas I spend my time developing C++/C programs. We have an option as well: coredumps. When you load the corefile into your debugger you can inspect every variable in your application at the time you generated the coredump, and you can look at the callstacks from all of the threads in your program. Personally I find it much more fun to use the debugger to inspect the corefile instead of reading through miles of logfiles...

With this in mind, rethink the excuse with timing issues. Wouldn't it be better to just dump core when you encounter the problem and load the corefile into your favorite debugger :-) .

You may think that dumping core is brutal to your users, because not all failures are fatal errors. You may be able to recover gracefully from some of the errors, but even if you don't know what leads to the error I still generate a coredump so I can dig into the problem. To avoid shutting down the service, I'll just fork off a copy of the program to generate a dump from:

#define recoverable_assert(ev) do_recoverable_assert(ev, #ev, __FILE__, __LINE__)

int do_recoverable_assert(int eval, const char *expression, const char *file, int lineno)
{
   if (eval == 0) {
      if (fork() == 0) {
         fprintf(stderr, "%s:%u: %s\n", file, line, expression);
         abort();
      }
      return 1;
   }

   return 0;
}

... cut ...

if (recoverable_assert((address % 8) == 0) {
   /* the address for the client buffer isn't aligned, start recover */
}

Unless you have modified your environment, you should get coredumps when your program performs an illegal operation. Unfortunately some engineers/managers think it's inappropriate for the user to get a coredump from a program, so they add logic into their program to trap such signals and exit cleanly. Personally I don't think that this is a good idea because the engineers lose valuable information when a problem occurs at a customer's site. With the corefile available you may investigate on a problem you fail to reproduce locally (and if the customer don't want to release the corefile due to security reasons, it is still possible to debug on-site). If the user doesn't want the coredump, they should turn this feature off in his shell/startup script before starting the program.

Posted at 11:13AM Sep 11, 2009 by trond in OpenSolaris  |  Comments[2]

Support for Monotone VCS

So today I learned about yet another VCS when someone requested support for Monotone. OpenGrok already supports Git, Mercurial, Bazaar, SCCS, Teamware, CVS, RCS, Subversion, Perforce, ClearCase and Razor, so adding yet another system was a trivial thing to do. You need to build OpenGrok from source if you want to try it out right now, or wait for 0.8 to be released :-)

Posted at 03:31PM Aug 28, 2009 by trond in OpenGrok  |  Comments[1]

Callback based protocol parser in libmemcached?

I have been working on designing a small library for memcached protocol handling. The intention of the library is to parse the memcached protocol for you, and create callbacks to your application so that you can implement the function. By using this library you should be able to add support for the memcached binary protocol to your application. Please note that the intention of this library is not to create a replacement / yet another fork of memcached!

Before I'm going ahead and spend time implementing the stuff I would like to agree upon an API. I don't care much about the ASCII protocol, so I added _binary_ in all the function nimes if anyone wants to implement something similar for the ASCII protocol.

So how does it look? Instead of just throwing my proposal for the API to you, I'll try to describe when and how each function is used. Please note that the current API proposal doesn't contain any factory methods where you can specify the memory area to use (to avoid calling malloc). Why? well I don't want waste the best function names for such methods because I don't see using that version of the factory methods as being the main usage of the library.

The first thing you need to do in your application is to create a handle to the protocol handler. To avoid locking inside the library you can _only_ access the protocol handler from one thread at a time (unless you add synchronization yourself). If you want to run with multiple threads you should let each thread have its own instance of the protocol handler instead. The function looks like:

/**
* Create and initialize an instance of the protocol handler.
* Please note that the library does not copy the
* callback structure, and you may use the same callback structure
* for all of your instances of the library. You must
* ensure that the memory is valid throughout the use of the instances.
*
* @param callback The callbacks to use from this protocol handler.
* @return NULL if allocation of an instance fails
*/
LIBMEMCACHED_API
struct memcached_binary_protocol_st *memcached_binary_protocol_create_instance(struct memcached_binary_protocol_callback_st *callback);

We will get back to a description of the callback structure later on. You release the instance when you are done using the library with the following function:

/**
* Destroy an instance of the protocol handler
*
* @param instance The instance to destroy
*/
LIBMEMCACHED_API
void memcached_binary_protocol_destroy_instance(struct memcached_binary_protocol_st *instance);

With a handle to the protocol library you can listen to a server socket and accept new clients. When a new client connects to the socket, you need to create a client structure and associate it with the socket:

/**
* Create a new client instance and associate it with a socket
* @param instance the protocol instance to bind the client to
* @param sock the client socket
* @return NULL if allocation fails, otherwise an instance
*/
LIBMEMCACHED_API
struct memcached_binary_protocol_client_st *memcached_binary_protocol_create_client(struct memcached_binary_protocol_st *instance, int sock);

With the client connection in hand, we can tell the protocol library to start to work on the client by calling:

enum MEMCACHED_BINARY_PROTOCOL_EVENT { ERROR_EVENT, READ_EVENT, WRITE_EVENT, READ_WRITE_EVENT };

/**
* Let the client do some work. This might involve reading / sending data
* to/from the client, or perform callbacks to execute a command.
* @param client the client structure to work on
* @return The next event the protocol handler will be notified for
*/
LIBMEMCACHED_API
enum MEMCACHED_BINARY_PROTOCOL_EVENT memcached_binary_protocol_client_work(struct memcached_binary_protocol_client_st *client);

This function will try to read data from the network and fire the callbacks with the given commands, and return the events it is interested of being notified on. If ERROR_EVENT is returned you should close the socket and destroy the client handle with:

/**
* Destroy a client handle.
* The caller needs to close the socket accociated with the client
* before calling this function. This function invalidates the
* client memory area.
*
* @param client the client to destroy
*/
LIBMEMCACHED_API
void memcached_binary_protocol_client_destroy(struct memcached_binary_protocol_client_st *client);

That's all you _need_ to know, but there is also some utility functions:

/**
* Get the socket attached to a client handle
* @param client the client to query
* @return the socket handle
*/
LIBMEMCACHED_API
int memcached_binary_protocol_client_get_socket(struct memcached_binary_protocol_client_st *client);

/**
* Get the error id socket attached to a client handle
* @param client the client to query for an error code
* @return the OS error code from the client
*/
LIBMEMCACHED_API
int memcached_binary_protocol_client_get_errno(struct memcached_binary_protocol_client_st *client);

Earlier I told you that I would come back to the callback structures, so let's start describing them. The memcached_binary_protocol_callback_st is the _only_ structure in the protocol handler library you are allowed to touch the internals of, and it is used to specify the callbacks you are interested in:

struct memcached_binary_protocol_callback_st {
 /**
  * The interface version used (set to 0 if you don't have any specialized
  * command handlers).
  */
 uint64_t interface_version;

 /**
  * Callback fired just before the command will be executed.
  *
  * @param cookie id of the client receiving the command
  * @param header the command header as received on the wire. If you look
  *               at the content you must ensure that you don't
  *               try to access beyond the end of the message.
  */
 void (*pre_execute)(const void *cookie,
                     protocol_binary_request_header *header);
 /**
  * Callback fired just after the command was exected (please note
  * that the data transfer back to the client is not finished at this
  * time).
  *
  * @param cookie id of the client receiving the command
  * @param header the command header as received on the wire. If you look
  *               at the content you must ensure that you don't
  *               try to access beyond the end of the message.
  */
 void (*post_execute)(const void *cookie,
                      protocol_binary_request_header *header);

 /**
  * Callback fired if no specialized callback is registered for this
  * specific command code.
  *
  * @param cookie id of the client receiving the command
  * @param header the command header as received on the wire. You must
  *               ensure that you don't try to access beyond the end of the
  *               message.
  * @param response_handler The response handler to send data back.
  */
 protocol_binary_response_status (*unknown)(const void *cookie,
                                            protocol_binary_request_header *header,
                                            memcached_binary_protocol_response_handler response_handler);

 /**
  * The different interface levels we support. A pointer is used so the
  * size of the structure is fixed. You must ensure that the memory area
  * passed as the pointer is valid as long as you use the protocol handler.
  */
 union {
    /**
     * The first version of the callback struct containing all of the
     * documented commands in the initial release of the binary protocol
     * (aka. memcached 1.4.0).
     */
    struct memcached_binary_protocol_callback_v1_st *v1;
 } interface;
};

The memcached_binary_protocol_response_handler is a function you need to call to send data back to the client:

/**
* Each command-callback will supply a response-handler so that you can
* send data back to the client.
*
* @param cookie Just pass along the cookie supplied in the callback
* @param status The status code for your reply (see protocol_binary.h)
*               for legal values.
* @param key What to insert as key in the reply (may be NIL)
* @param keylen The length of the key (should be 0 if key is NIL)
* @param body What to store in the body of the package (may be NIL)
* @param bodylen The number of bytes of the body (should be 0 if
*                body is NIL)
* @param cas The CAS value to insert into the response (should be 0
*            if you don't care)
* @param datatype Should be PROTOCOL_BINARY_RAW_BYTES
*
*/
typedef void (*memcached_binary_protocol_response_handler)(const void *cookie,
                                                         protocol_binary_response_status status,
                                                         const void *key,
                                                         uint16_t keylen,
                                                         const void *body,
                                                         uint32_t bodylen,
                                                         uint64_t cas,
                                                         protocol_binary_datatypes datatype);

So the simplest example for you would be:

static struct memcached_binary_protocol_callback_st callback= { .unknown= my_function_callback; };

struct memcached_binary_protocol_st *handle;
handle= memcached_binary_protocol_create_instance(&callback);

It wouldn't help you much if you had to read the spec to get all the juicy details on how the protocol looks for all of the different commands, and thats what the interface-union is used for. My proposal for v1 looks like:

/**
* The first version of the callback struct containing all of the
* documented commands in the initial release of the binary protocol
* (aka. memcached 1.4.0).
*
* You might miss the Q commands (addq etc) but the response function
* knows how to deal with them so you don't need to worry about that :-)
*/
struct memcached_binary_protocol_callback_v1_st {
 /**
  * Add an item to the cache
  * @param cookie id of the client receiving the command
  * @param key the key to add
  * @param len the length of the key
  * @param val the value to store for the key (may be NIL)
  * @param vallen the length of the data
  * @param flags the flags to store with the key
  * @param exptime the expiry time for the key-value pair
  * @param response_handler to send the result back to the client.
  */
 protocol_binary_response_status (*add)(const void *cookie,
                                        const void *key,
                                        uint16_t keylen,
                                        const void* val,
                                        uint32_t vallen,
                                        uint32_t flags,
                                        uint32_t exptime,
                                        memcached_binary_protocol_response_handler response_handler);

 /**
  * Append data to an existing key-value pair.
  *
  * @param cookie id of the client receiving the command
  * @param key the key to add data to
  * @param len the length of the key
  * @param val the value to append to the value
  * @param vallen the length of the data
  * @param cas the CAS in the request
  * @param response_handler to send the result back to the client
  *
  */
 protocol_binary_response_status (*append)(const void *cookie,
                                           const void *key,
                                           uint16_t keylen,
                                           const void* val,
                                           uint32_t vallen,
                                           uint64_t cas,
                                           memcached_binary_protocol_response_handler response_handler);

 /**
  * Decrement the value for a key
  *
  * @param cookie id of the client receiving the command
  * @param key the key to decrement the value for
  * @param len the length of the key
  * @param delta the amount to decrement
  * @param initial initial value to store (if the key doesn't exist)
  * @param expiration expiration time for the object (if the key doesn't exist)
  * @param cas the CAS in the request
  * @param response_handler to send the result back to the client
  *
  */
 protocol_binary_response_status (*decrement)(const void *cookie,
                                              const void *key,
                                              uint16_t keylen,
                                              uint64_t delta,
                                              uint64_t initial,
                                              uint32_t expiration,
                                              memcached_binary_protocol_response_handler response_handler);

 /**
  * Delete an existing key
  *
  * @param cookie id of the client receiving the command
  * @param key the key to delete
  * @param len the length of the key
  * @param cas the CAS in the request
  * @param response_handler to send the result back to the client
  */
 protocol_binary_response_status (*delete)(const void *cookie,
                                           const void *key,
                                           uint16_t keylen,
                                           uint64_t cas,
                                           memcached_binary_protocol_response_handler response_handler);


 /**
  * Flush the cache
  *
  * @param cookie id of the client receiving the command
  * @param when when the cache should be flushed (0 == immediately)
  * @param response_handler to send the result back to the client
  */
 protocol_binary_response_status (*flush)(const void *cookie,
                                          uint32_t when,
                                          memcached_binary_protocol_response_handler response_handler);



 /**
  * Get a key-value pair
  *
  * @param cookie id of the client receiving the command
  * @param key the key to get
  * @param len the length of the key
  * @param response_handler to send the result back to the client
  */
 protocol_binary_response_status (*get)(const void *cookie,
                                        const void *key,
                                        uint16_t keylen,
                                        memcached_binary_protocol_response_handler response_handler);

 /**
  * Increment the value for a key
  *
  * @param cookie id of the client receiving the command
  * @param key the key to increment the value on
  * @param len the length of the key
  * @param delta the amount to increment
  * @param initial initial value to store (if the key doesn't exist)
  * @param expiration expiration time for the object (if the key doesn't exist)
  * @param cas the CAS in the request
  * @param response_handler to send the result back to the client
  *
  */
 protocol_binary_response_status (*increment)(const void *cookie,
                                              const void *key,
                                              uint16_t keylen,
                                              uint64_t delta,
                                              uint64_t initial,
                                              uint32_t expiration,
                                              memcached_binary_protocol_response_handler response_handler);

 /**
  * The noop command was received. This is just a notification callback (the
  * response is automatically created).
  *
  * @param cookie id of the client receiving the command
  */
 protocol_binary_response_status (*noop)(const void *cookie);

 /**
  * Prepend data to an existing key-value pair.
  *
  * @param cookie id of the client receiving the command
  * @param key the key to prepend data to
  * @param len the length of the key
  * @param val the value to prepend to the value
  * @param vallen the length of the data
  * @param cas the CAS in the request
  * @param response_handler to send the result back to the client
  *
  */
 protocol_binary_response_status (*prepend)(const void *cookie,
                                            const void *key,
                                            uint16_t keylen,
                                            const void* val,
                                            uint32_t vallen,
                                            uint64_t cas,
                                            memcached_binary_protocol_response_handler response_handler);

 /**
  * The quit command was received. This is just a notification callback (the
  * response is automatically created).
  *
  * @param cookie id of the client receiving the command
  */
 protocol_binary_response_status (*quit)(const void *cookie);


 /**
  * Replace an existing item to the cache
  *
  * @param cookie id of the client receiving the command
  * @param key the key to replace the content for
  * @param len the length of the key
  * @param val the value to store for the key (may be NIL)
  * @param vallen the length of the data
  * @param flags the flags to store with the key
  * @param exptime the expiry time for the key-value pair
  * @param cas the cas id in the request
  * @param response_handler to send the result back to the client.
  */
 protocol_binary_response_status (*replace)(const void *cookie,
                                            const void *key,
                                            uint16_t keylen,
                                            const void* val,
                                            uint32_t vallen,
                                            uint32_t flags,
                                            uint32_t exptime,
                                            uint64_t cas,
                                            memcached_binary_protocol_response_handler response_handler);


 /**
  * Set a key-value pair in the cache
  *
  * @param cookie id of the client receiving the command
  * @param key the key to insert
  * @param len the length of the key
  * @param val the value to store for the key (may be NIL)
  * @param vallen the length of the data
  * @param flags the flags to store with the key
  * @param exptime the expiry time for the key-value pair
  * @param response_handler to send the result back to the client.
  */
 protocol_binary_response_status (*set)(const void *cookie,
                                        const void *key,
                                        uint16_t keylen,
                                        const void* val,
                                        uint32_t vallen,
                                        uint32_t flags,
                                        uint32_t exptime,
                                        memcached_binary_protocol_response_handler response_handler);

 /**
  * Get status information
  *
  * @param cookie id of the client receiving the command
  * @param key the key to get status for (or NIL to request all status).
  *            Remember to insert the terminating packet if multiple
  *            packets should be returned.
  * @param keylen the length of the key
  * @param response_handler to send the result back to the client, but
  *                         don't send reply on success!
  *
  */
 protocol_binary_response_status (*stat)(const void *cookie,
                                         const void *key,
                                         uint16_t keylen,
                                         memcached_binary_protocol_response_handler response_handler);

 /**
  * Get the version information
  *
  * @param cookie id of the client receiving the command
  * @param response_handler to send the result back to the client, but
  *                         don't send reply on success!
  *
  */
 protocol_binary_response_status (*version)(const void *cookie,
                                            memcached_binary_protocol_response_handler response_handler);
};

Comments?

Posted at 03:08PM Aug 10, 2009 by trond in Memcached  |  Comments[0]

Hudson Bazaar plugin released

I released the first version of my Bazaar plugin for Hudson a few days ago, so you should be able to install it by logging into your Hudson server and select: Manage Hudson, Manage Plugins, Available.

To use the Bazaar plugin, press Configure for the desired project and scroll down to the Source Code Management section and select Bazaar. Fill in the Repository URL, and if you would like to nuke the repository each time, press the advanced button and check the "Clean build" check-box. See:

Posted at 01:19PM Aug 10, 2009 by trond in OpenSolaris  |  Comments[0]

Back from Vacation

I just got back from a 3 week vacation visiting museums and amusement parks in Oslo with the kids. We also visited Geiranger to attend a wedding, and what a wedding! It started with a wonderful ceremony at the church, before we went on a 3 hour boat trip on the fjords from the church all the way to Geiranger. It was really incredible to see De syv søstre, Friaren and all of the mountains from the deck. It was absolutely fantastic! Just look at the view from our hotel room!

Posted at 09:04PM Aug 03, 2009 by trond in Personal  |  Comments[0]

Memcached 1.4.0 released!

We released memcached 1.4.0 earlier today! It is more than two years since we started the work on one of the most important part of this release, the binary protocol. I guess a lot of users doesn't really care about the binary protocol, but it makes the implementation of other features easier (the replication in libmemcached is only available if you use the binary protocol).

Other highlights in 1.4.0:

• Performance improvements (removed lock contention)
• Run without privileges on Solaris
• More statistics information

Check out the full release notes at http://code.google.com/p/memcached/wiki/ReleaseNotes140.

Posted at 12:18PM Jul 10, 2009 by trond in Memcached  |  Comments[3]

Scale beyond 8 cores?

If you try to benchmark a memcached server, you will see that it scales relatively good up to 4 threads, and if you go beyond 8 threads the throughput will start to drop. People have been talking about the scalability problems of the memcached server on the mailing lists, IRC and on the hackathons with various solutions to the problems. If you look at the implementation of the memcached server, you will see that it use only a limited set of locks:

Lock	Purpose
slabs_lock	The internal memory management of memcached use one single lock to protect access to the internal memory allocator. People tend to want to split this mutex into a separate mutex per slab class, but I don't think that this will give you any measurable performance wins. Why? well the _only_ time we will try to access this lock is during an operation that will modify an entry in the cache (add, set, replace, append, prepend, incr, decr). If this lock really is a performance bottleneck, your GET load is lower than your SET load and that is not the typical use pattern of memcached (and I would like to optimize for the common case...)
stats_lock	In 1.2.x all modifications of statistical information was protected by this single lock. plockstat revealed that we had mutex contention on this mutex, so in 1.4 most of the statistics are collected per thread, and aggregated when you call the stats command.
conn_lock	Access to the list of connection structures is guarded by this lock. If this lock comes up when you try to benchmark your memcached server, you would be way better off by reusing your connections to the memcached server.
cache_lock	All access to the internal hash table is protected by this lock, and this is the lock I'm going to talk a bit more about!

If you look at the implementation of the memcached server, it stores all of the items in one large hash map. Every time we need to insert, delete or search the hash table, we will try to get exclusive access to the entire hash table. And this is pretty much all that memcached does ;-) (You send a request, it looks up the item, and sends it back to you).

So how can we easily fix this problem without rewriting everyting? Well we could partition up the hash into multiple partitions and only lock down a subset of the hash instead of the complete hash. To avoid extra locking to update the LRU (if it spans two partitions) list I decided to let each partition have it's own LRU list. This sounded like a pretty easy fix to implement, so I just grabbed my laptop and implemented it while watching C.S.I on TV with my girlfriend one night :-)

So how did it scale? Zoran Radovic wrote the following blog entry: Scaling Memcached: 500,000+ Operations/Second with a Single-Socket UltraSPARC T2. If you are interested in the code, you can get it from http://github.com/trondn/memcached/tree/partition.

Posted at 01:10PM Jul 06, 2009 by trond in Memcached  |  Comments[5]