Trond Norbye's Weblog

memcapable

Earlier today Matt Ingenthron blogged about the new tool memcapable I wrote a while back. In his blog Matt mentions some of the reasons why we want such a tool, but he didn't actually mention the reason for why I actually sat down to create the tool.

If you follow my blog you might remember my entry "Callback based protocol parser in libmemcached?". Before I could start implementing the parser, I really needed a tool to:

1. send all of the defined packet structures to the server
2. verify the response packet generated from the server

I didn't have the need for the textual protocol at the time I wrote the initial version of memcapable, so right now memcapable can only be used to test the binary protocol (not all variants of all commands are implemented in the initial version).

So how does it work? In it's simplest form you can use it to test the memcached server running on the default port on the same computer:

trond@storm> ./memcapable
noop		[pass]
quit		[pass]
quitq		[pass]
set		[pass]
setq		[pass]
flush		[pass]
flushq		[pass]
add		[pass]
addq		[pass]
replace		[pass]
replaceq		[pass]
delete		[pass]
deleteq		[pass]
get		[pass]
getq		[pass]
getk		[pass]
getkq		[pass]
incr		[pass]
incrq		[pass]
decr		[pass]
decrq		[pass]
version		[pass]
append		[pass]
appendq		[pass]
prepend		[pass]
prependq		[pass]
stat		[pass]
illegal		[pass]
All tests passed

Now this looks really nice doesn't it, but let's try to run it on a 1.2.8 version:

trond@storm> ./memcapable
noop		[FAIL]
quit		[FAIL]
quitq		[FAIL]
set		[FAIL]
setq		[FAIL]
flush		[FAIL]
flushq		[FAIL]
add		[FAIL]
addq		[FAIL]
replace		[FAIL]
replaceq		[FAIL]
delete		[FAIL]
deleteq		[FAIL]
get		[FAIL]
getq		[FAIL]
getk		[FAIL]
getkq		[FAIL]
incr		[FAIL]
incrq		[FAIL]
decr		[FAIL]
decrq		[FAIL]
version		[FAIL]
append		[FAIL]
appendq		[FAIL]
prepend		[FAIL]
prependq		[FAIL]
stat		[FAIL]
illegal		[FAIL]
28 of 28 tests failed

This shouldn't come as a big surprise, because the binary protocol isn't implemented in 1.2.8. Getting [FAIL] isn't really that informative, because it doesn't help you as a developer to figure out what's wrong. I have added a couple of options to the program that may help you to track down the real problem: -v and -c.

-v

Print out the assertion that failed

-c

Create a coredump when an assertion fails

Let's start the memcached server and disable the use of CAS and re-run memcapable with -v -c

trond@storm> ./memcapable -v -c
noop		[pass]
quit		[pass]
quitq		[pass]
set		memcapable.c:493: rsp->plain.message.header.response.cas != 0
zsh: IOT instruction (core dumped)  ./memcapable -v -c

As you can see it expects the response packet to have a CAS value set for the operation. If you would like to inspect the response packet you could load it into your debugger and poke around:

trond@storm> dbx - core
Corefile specified executable: "/source/libmemcached/memcapable/clients/memcapable"
Reading memcapable
core file header read successfully
Reading ld.so.1
Reading libm.so.2
Reading libnsl.so.1
Reading libsocket.so.1
Reading libpthread.so.1
Reading libthread.so.1
Reading libc.so.1
t@1 (l@1) program terminated by signal ABRT (Abort)
0xfffffd7fff2842aa: _lwp_kill+0x000a:	jae      _lwp_kill+0x18	[ 0xfffffd7fff2842b8, .+0xe ]
Current function is ensure
  212         abort();
(dbx) where
current thread: t@1
  [1] _lwp_kill(0x1, 0x6, 0xffffff01cdf92ae0, 0xfffffd7fff284c0e, 0x12, 0x0), at 0xfffffd7fff2842aa 
  [2] thr_kill(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff2788cd 
  [3] raise(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff227511 
  [4] abort(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff1fda41 
=>[5] ensure(val = 0, expression = 0x408498 "rsp->plain.message.header.response.cas != 0", file = 0x408198 "memcapable.c", line = 493), line 212 in "memcapable.c"
  [6] do_validate_response_header(rsp = 0xfffffd7fffdfeed8, cc = '\001', status = 0), line 493 in "memcapable.c"
  [7] test_binary_set_impl(key = 0x4087b0 "test_binary_set", cc = '\001'), line 621 in "memcapable.c"
  [8] test_binary_set(), line 651 in "memcapable.c"
  [9] main(argc = 3, argv = 0xfffffd7fffdff798), line 1196 in "memcapable.c"
(dbx) frame 6
Current function is do_validate_response_header
  493         verify(rsp->plain.message.header.response.cas != 0);
(dbx) print rsp->plain.message.header.response
rsp->plain.message.header.response = {
    magic    = '�'
    opcode   = '\001'
    keylen   = 0
    extlen   = '\0'
    datatype = '\0'
    status   = 0
    bodylen  = 0
    opaque   = 3735928559U
    cas      = 0
}

I noticed earlier today that there are some issues with the -v flag if you don't include -c (it just prints out the assertion and reports everything back up as success ;-)). I'll try to address that in the next release.

Posted at 11:39PM Sep 21, 2009 by trond in Memcached  |  Comments[0]