Apr 232013
 

Recently, I wrote a post about using iptables to block certain IPs from accessing my server, making it drop traffic without acknowledgement, so that I would appear to no longer exist on the internet. My intent was to mitigate a small brute force login problem I was seeing on my server from one IP address. However, since then, I’ve learned from a variety of sources (including Ars Technica and pretty much every other electronic news site) that this was likely just a small window into a much larger botnet attack that is attempting to compromise wordpress installs using many machines at once. Obviously, in this case, manually blacklisting each IP address is infeasible. I recommended installing Limit Login Attempts as a good plugin to prevent botnets from attempting to brute force one’s admin page, but now it is time to go a step further. I would like to expand the blacklisting to simply drop traffic from anyone who exceeds the login limit, in order to save my server the processing time for handling illegitimate requests.

In order to do this, I’m going to be using a program known as fail2ban which monitors log files for pre-specified regular expressions that identify undesirable behavior, and then blocks traffic from the IP addresses generating this behavior. The bans are timed and are added and removed automatically by the fail2ban daemon program, with configuration options for the duration, number of failures necessary to trigger a ban, etc. By combining fail2ban with Limit Login Attempts, we can leverage both a WordPress-level solution to identify problem users easily and a kernel firewall-level solution to remove them from the server with minimal impact.

Fail2ban operates on log file parsing, which means it is incompatible with currently released versions of Limit Login Attempts. I will talk to the author and see if I can get him to incorporate writing to an actual log file in addition to logging in the MySQL database running wordpress, but for now, we’ll simply hack in the functionality we need, because it only takes a few lines. In limit-login-attempts.php, in the function limit_login_notify_log, which starts around line 563, we’ll append the following lines to the end of the function, making it write a simple message with the user’s IP address to a pre-specified and hardcoded log file name whenever a lockout is also written to the database:

$file = fopen(LIMIT_LOGIN_FILE_PATH, 'at');
if ($file) {
// Note that the 't' flag will offer \n to \r\n translation on windows
fwrite($file, date('M j h:i:s ') . sprintf(__('IP locked out by wp-limit-login: %s', 'limit-login-attempts'), $ip) . "\n");
fclose($file);
}

Then, we also need to add a line defining the constant LIMIT_LOGIN_FILE_PATH, so I’ve added define(LIMIT_LOGIN_FILE_PATH, ‘/var/log/nginx/wp-limit-login.log’); at the top of the file. Perfect, now whenever logging is enabled (and why would you disable it?), the offending user’s information will be written to both the MySQL database to display in the Limit Login Attempts dashboard and to the specified log file. Make sure the file path you choose for the log file is writeable by whichever process runs your php instance (be it apache, php-fpm, etc). This log file is very simple, which suits my purposes, but you may want to augment yours with some other information, such as potentially the name of the blog, if you’re running a multi-site installation, or the number of times an IP has been locked out. It’s also entirely possible to put this file inside your wordpress install, if you don’t have access to /var/log for instance. If you put it inside the plugin directory itself, the standard wordpress rewrite rules should prevent clients from accessing it, but I haven’t verified this, so I’d recommend keeping it outside of your webroot to avoid disclosing the IP addresses (we’re responsible administrators, even if the people we’re banning aren’t).

Next, we add a filter to the fail2ban configuration. Filters are defined in /etc/fail2ban/filter.d, one per file. I suggest a simple name, like wplogin.conf. Inside this filter, we configure the regular expression to capture a failure and an optional regular expression to ignore, which trumps a failure line. Based on the output above, our filter is very simple:

# Capture failed login attempts through wordpress limit logins module

[Definition]

# default message is "IP locked out by wp-limit-login: <ip address>"
failregex = [^:]*: <HOST>

ignoreregex =

With the filter in place, we must add a jail configuration to fail2ban. For that, we edit /etc/fail2ban/jail.conf and add a new jail that references the filter we just created. I would strongly recommend that you add your own IP address on the ignoreip line , in CIDR notation, because if not, all of your traffic to your server will be dropped (for a specified amount of time), which will make it impossible for you to access.

[wplogin]
enabled = true
filter = wplogin
action = iptables-allports[name=wplogin]
logpath = /var/log/nginx/wp-limit-login.log
maxretry = 1
bantime = 600
ignoreip = 127.0.0.1/32

Because Limit Login Attempts already counts how many times someone has failed to log in, an entry here means they should be banned. Therefore, we set the retry count to just one, so that it will ban them on their first attempt. We also use the iptables-allports action to ban the user’s IP address, which does almost exactly the same thing as the script I presented before, except that it uses a separate fail2ban chain and names the rules so that it can go back and remove them later with minimal fuss (i.e. it is more sophisticated). I would also recommend increasing the ban time, as the default is only 10 minutes. Unless you run into problems with actual readers being banned (unlikely, as the botnet appears to run on other servers, not on infected clients), you can easily set this to an hour, or a day, at your discretion.

After that, simple launch fail2ban by executing:

# fail2ban-client start

and fail2ban will begin monitoring the log file. You may notice that fail2ban gives a warning on startup saying that the log file doesn’t exist. If this happens, create the file, change its owner to match the account that will be writing to it, and restart fail2ban:

# touch /var/log/nginx/wp-limit-login.log
# chown nginx /var/log/nginx/wp-limit-login.log
# fail2ban-client reload

This should combine the power of Limit Login Attempts to protect your installation from brute force password attacks and fail2ban to drop traffic to your server, improving its effectiveness. The combination will reduce the amount of server resources spent dealing with illegitimate requests, which can turn an almost-DoS caused by brute force password hammering into a non-issue.

Apr 022013
 

Yesterday, I had a small issue where someone attempted to brute force the admin password to my blog, resulting in significantly decreased availability for about 10 minutes as my server’s resources were maxed out, leading to 45+ second page load times. Luckily, it happened just as I was coming home, so I was able to identify the problem and put a stop to it very quickly. Incidentally, this taught me that wordpress does not have a rate limiting algorithm in place on logins (allowing brute force attacks at the server’s maximum request speed). So, I would recommend using a plugin to add rate limiting to logins, which can mitigate the danger associated with a simple scripted attack.

Once the problem was identified as an attack of some kind, the next step was to disable php-fpm in order to drop the load average to an acceptable level so that I could figure out who and what was going on, and then do my best to mitigate it. Incidentally, this will cause nginx to produce a 502 error as the CGI handler is no longer responsive. From /var/log/nginx/blog.access.log:

{IP} - - [01/Apr/2013:16:59:57 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:57 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

becomes

{IP} - - [01/Apr/2013:22:45:53 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Interestingly, we can see that whatever brute force attack is being used attempts to spoof the bing search crawling robot in order to prevent me from using a user-agent denial without deindexing myself. This isn’t really a big deal, because I don’t plan on solving the problem with policies in nginx, so anything in the http request is largely irrelevant.

I’ve removed the user’s IP address, but in this case it was assigned through a cloud services provider, meaning that someone likely set up an account to look for wordpress blogs on the internet and attempt to break into their admin directory, likely to install a plugin and then take over the machine in order to attach it to a botnet. I’ve emailed their abuse department, but I never heard back and don’t really expect them to do anything about it (most companies don’t seem to care). I don’t expect any of my traffic to come from cloud providers, so the simplest solution is to deny all traffic from the IP in question. There are a number of ways for me to do this, but the best is to drop requests as early as possible in my network stack, because it minimizes the amount of resources that my server will dedicate to processing an illegitimate request. To do this, I use the kernel firewall utility iptables, adding rules to drop all incoming packets (with no acknowledgement at all, which makes it appear as though my server is no longer online) from blacklisted sources as early as possible in the processing chain.

General scanning of the internet, probing for vulnerabilities, and attempting to exploit poorly configured servers is extremely common. I used to get very concerned when I saw logfile entries almost daily indicating that someone had attempted to request files that don’t exist or exploit a flaw in how older versions of apache (which I don’t use) looks up files. After doing some reading, I relaxed, because it’s very common, and servers running up-to-date software will withstand generic attacks (but likely not actual, targeted attempts at breaching). That said, since I have no interest in letting these people continue to scan me for vulnerabilities in order to make me a part of their botnet, and they almost certainly do not form any part of my readership, I put together a small script that added a log message and dropped the packets whenever an IP I disliked attempted to reach the server, to make it quick and easy to deny people whose nonsense was becoming problematic.

#!/bin/bash

if [ -z $1 ]; then
        echo "Usage: $0 <ip address> [reason]";
        exit;
fi

IP=$1
REASON="denied ip"

if [ $# -gt 1 ]; then
        shift;
        REASON=$@
fi

iptables -A INPUT -s ${IP}/32 -j LOG --log-prefix "${REASON} "
iptables -A INPUT -s ${IP}/32 -j DROP

You can see log messages (including those generated by iptables) in /var/log/messages where you will see a message for each dropped packet. This could potentially cause your log file to balloon, in the case of any kind of flooding, so you may wish to eliminate the log rules, but I generally keep them so that I remember later why an IP address was blocked.

Feel free to use this whenever you find you need to deny access to a specific IP address, to save the trouble of learning iptables’ options. If you add lots of entries to your INPUT chain in iptables, all of your network processing will slow down, as each incoming packet is matched against all of the rules in the chain in order until it reaches the end, where it is passed up to the next layer. That said, the slowdown is relatively minor and does not become noticeable until you have added hundreds of rules. If you find yourself in this situation, perhaps investigate another approach, or consider using multiple chains and branching as early as possible, rather than listing all blacklisted IPs sequentially, to limit the maximum chain traversal length.