I was got out of bed with an emergency call from a client at the weekend because his blog was down. This is the one that is getting a lot of traffic, but CPU was spiking up to 100% quite unnecessarily. Apache was running about 50 process on ‘ps aux’, each using a few % of CPU. It was very ugly.
First thing was the Apache error log showed lots of “Unable to allocate memory for pool” errors. The solution was found here and involved a couple of additions to /etc/php5/apache2/php.ini
apc.ttl=0 apc.shm_size=64M
I had a look through the logs and found nothing untoward and so put it down to APC throwing a wobbler, and something to look into in the morning. CPU load had settled to normal levels and things looked good.
Thirty minutes later, I was out of bed again with another phone call. This time, the problem was a bit more obvious from the Apache error and access logs. There were a few thousand of these:
68.106.150.133 - - [16/Mar/2012:18:50:52 +0000] "GET /{page}/?replytocom=8325 HTTP/1.1" 200 132607 "http://{domain}/{page}" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
Each one had an incremented value for ‘replytocom’ so it was clearly scripted. And these only became obvious when I had reduced MaxClients to 10 which gave Apache chance to serve every request, rather than just choking. Definitely something I’d do again under the same circumstance, as I missed this the first time.
Previously I’ve only ever used the Amazon Security Groups to provide firewalling. But after a bit of searching, I found these ufw commands that did the trick:
sudo ufw deny from 68.106.150.133 #Block the attacker sudo ufw allow 80 #Allow everyone else to hit the webserver sudo ufw allow 1022 #This port is used for SSH sudo ufw enable #Enable the firewall
It’s not ideal, especially if the attacker is on a dynamic IP and can just reset their modem, and start again. But at that time, it was enough. And abuse@cox.net got an email too.
Tomorrow I’ll have to look at better fixes. I make use of Fail2Ban but have found it doesn’t cope well with monitoring access logs on high traffic sites in the past. But that was when using a regex, and a filter for DoS might not need that, so it might be worth a try. Otherwise it looks like Apache mod_evasive is the way to go. With either of these, the tricky part will be balancing the filters to avoid banning real users whilst still catching the bad guys.
0 Comments.