The Lone Pole

Yesterday, I had a small issue where someone attempted to brute force the admin password to my blog, resulting in significantly decreased availability for about 10 minutes as my server's resources were maxed out, leading to 45+ second page load times. Luckily, it happened just as I was coming home, so I was able to identify the problem and put a stop to it very quickly. Incidentally, this taught me that wordpress does not have a rate limiting algorithm in place on logins (allowing brute force attacks at the server's maximum request speed). So, I would recommend using a plugin to add rate limiting to logins, which can mitigate the danger associated with a simple scripted attack.

Once the problem was identified as an attack of some kind, the next step was to disable php-fpm in order to drop the load average to an acceptable level so that I could figure out who and what was going on, and then do my best to mitigate it. Incidentally, this will cause nginx to produce a 502 error as the CGI handler is no longer responsive. From /var/log/nginx/blog.access.log:

{IP} - - [01/Apr/2013:16:59:57 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:57 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

becomes

{IP} - - [01/Apr/2013:22:45:53 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Interestingly, we can see that whatever brute force attack is being used attempts to spoof the bing search crawling robot in order to prevent me from using a user-agent denial without deindexing myself. This isn't really a big deal, because I don't plan on solving the problem with policies in nginx, so anything in the http request is largely irrelevant.

I've removed the user's IP address, but in this case it was assigned through a cloud services provider, meaning that someone likely set up an account to look for wordpress blogs on the internet and attempt to break into their admin directory, likely to install a plugin and then take over the machine in order to attach it to a botnet. I've emailed their abuse department, but I never heard back and don't really expect them to do anything about it (most companies don't seem to care). I don't expect any of my traffic to come from cloud providers, so the simplest solution is to deny all traffic from the IP in question. There are a number of ways for me to do this, but the best is to drop requests as early as possible in my network stack, because it minimizes the amount of resources that my server will dedicate to processing an illegitimate request. To do this, I use the kernel firewall utility iptables, adding rules to drop all incoming packets (with no acknowledgement at all, which makes it appear as though my server is no longer online) from blacklisted sources as early as possible in the processing chain.

General scanning of the internet, probing for vulnerabilities, and attempting to exploit poorly configured servers is extremely common. I used to get very concerned when I saw logfile entries almost daily indicating that someone had attempted to request files that don't exist or exploit a flaw in how older versions of apache (which I don't use) looks up files. After doing some reading, I relaxed, because it's very common, and servers running up-to-date software will withstand generic attacks (but likely not actual, targeted attempts at breaching). That said, since I have no interest in letting these people continue to scan me for vulnerabilities in order to make me a part of their botnet, and they almost certainly do not form any part of my readership, I put together a small script that added a log message and dropped the packets whenever an IP I disliked attempted to reach the server, to make it quick and easy to deny people whose nonsense was becoming problematic.

#!/bin/bash

if [ -z $1 ]; then
        echo "Usage: $0 <ip address> [reason]";
        exit;
fi

IP=$1
REASON="denied ip"

if [ $# -gt 1 ]; then
        shift;
        REASON=$@
fi

iptables -A INPUT -s {REASON} "
iptables -A INPUT -s ${IP}/32 -j DROP

You can see log messages (including those generated by iptables) in /var/log/messages where you will see a message for each dropped packet. This could potentially cause your log file to balloon, in the case of any kind of flooding, so you may wish to eliminate the log rules, but I generally keep them so that I remember later why an IP address was blocked.

Feel free to use this whenever you find you need to deny access to a specific IP address, to save the trouble of learning iptables' options. If you add lots of entries to your INPUT chain in iptables, all of your network processing will slow down, as each incoming packet is matched against all of the rules in the chain in order until it reaches the end, where it is passed up to the next layer. That said, the slowdown is relatively minor and does not become noticeable until you have added hundreds of rules. If you find yourself in this situation, perhaps investigate another approach, or consider using multiple chains and branching as early as possible, rather than listing all blacklisted IPs sequentially, to limit the maximum chain traversal length.