DoS on Amazon EC2 instances running WordPress

I was got out of bed with an emergency call from a client at the weekend because his blog was down. This is the one that is getting a lot of traffic, but CPU was spiking up to 100% quite unnecessarily. Apache was running about 50 process on ‘ps aux’, each using a few % of CPU. It was very ugly.

First thing was the Apache error log showed lots of “Unable to allocate memory for pool” errors. The solution was found here and involved a couple of additions to /etc/php5/apache2/php.ini

apc.ttl=0
apc.shm_size=64M

I had a look through the logs and found nothing untoward and so put it down to APC throwing a wobbler, and something to look into in the morning. CPU load had settled to normal levels and things looked good.

Thirty minutes later, I was out of bed again with another phone call. This time, the problem was a bit more obvious from the Apache error and access logs. There were a few thousand of these:

68.106.150.133 - - [16/Mar/2012:18:50:52 +0000] "GET /{page}/?replytocom=8325 HTTP/1.1" 200 132607 "http://{domain}/{page}" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" 

Each one had an incremented value for ‘replytocom’ so it was clearly scripted. And these only became obvious when I had reduced MaxClients to 10 which gave Apache chance to serve every request, rather than just choking. Definitely something I’d do again under the same circumstance, as I missed this the first time.

Previously I’ve only ever used the Amazon Security Groups to provide firewalling. But after a bit of searching, I found these ufw commands that did the trick:

sudo ufw deny from 68.106.150.133 #Block the attacker
sudo ufw allow 80 #Allow everyone else to hit the webserver
sudo ufw allow 1022 #This port is used for SSH
sudo ufw enable #Enable the firewall

It’s not ideal, especially if the attacker is on a dynamic IP and can just reset their modem, and start again. But at that time, it was enough. And abuse@cox.net got an email too.

Tomorrow I’ll have to look at better fixes. I make use of Fail2Ban but have found it doesn’t cope well with monitoring access logs on high traffic sites in the past. But that was when using a regex, and a filter for DoS might not need that, so it might be worth a try. Otherwise it looks like Apache mod_evasive is the way to go. With either of these, the tricky part will be balancing the filters to avoid banning real users whilst still catching the bad guys.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>