Memcached constant timeouts and high load

Just a rather silly one…

So the two webservers on one site started to run at a crawl. The timeout was so bad, the load balancer healthchecks were failing and they were up and down constantly. Lots of checking of webserver stuff showed nothing weird – no high load, no DoS in the webserver logs or netstat, no problems with memory.

On to the DB server, but again nothing wrong there. So next, on to the memcached server…

The problem here was obvious – a load of 2.00. But with memcached running at about 5% and nothing else happening, the reason for this wasn’t immediately clear. So a quick look at the memcached.log…which turned into a long look at the memcached,log because it refused to open. Well, not actually refused, but was so slow that I assume there was a problem.

And the problem was:

-rw-r--r--  1 root      root 6097502208 2012-04-04 03:56 memcached.log

Ooops. Quickly followed up with:

/var/log$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              8256952   7837516         8 100% /

Ah. So that’s what happens when the -vv super-verbose option is left on memcached after testing has been completed.

And after removing the log file, changing the config, and restarting memcached all is peachy once again.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>