Just a rather silly one…
So the two webservers on one site started to run at a crawl. The timeout was so bad, the load balancer healthchecks were failing and they were up and down constantly. Lots of checking of webserver stuff showed nothing weird – no high load, no DoS in the webserver logs or netstat, no problems with memory.
On to the DB server, but again nothing wrong there. So next, on to the memcached server…
The problem here was obvious – a load of 2.00. But with memcached running at about 5% and nothing else happening, the reason for this wasn’t immediately clear. So a quick look at the memcached.log…which turned into a long look at the memcached,log because it refused to open. Well, not actually refused, but was so slow that I assume there was a problem.
And the problem was:
-rw-r--r-- 1 root root 6097502208 2012-04-04 03:56 memcached.log
Ooops. Quickly followed up with:
/var/log$ df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 8256952 7837516 8 100% /
Ah. So that’s what happens when the -vv super-verbose option is left on memcached after testing has been completed.
And after removing the log file, changing the config, and restarting memcached all is peachy once again.
0 Comments.