Apr 232013
 

Recently, I wrote a post about using iptables to block certain IPs from accessing my server, making it drop traffic without acknowledgement, so that I would appear to no longer exist on the internet. My intent was to mitigate a small brute force login problem I was seeing on my server from one IP address. However, since then, I’ve learned from a variety of sources (including Ars Technica and pretty much every other electronic news site) that this was likely just a small window into a much larger botnet attack that is attempting to compromise wordpress installs using many machines at once. Obviously, in this case, manually blacklisting each IP address is infeasible. I recommended installing Limit Login Attempts as a good plugin to prevent botnets from attempting to brute force one’s admin page, but now it is time to go a step further. I would like to expand the blacklisting to simply drop traffic from anyone who exceeds the login limit, in order to save my server the processing time for handling illegitimate requests.

In order to do this, I’m going to be using a program known as fail2ban which monitors log files for pre-specified regular expressions that identify undesirable behavior, and then blocks traffic from the IP addresses generating this behavior. The bans are timed and are added and removed automatically by the fail2ban daemon program, with configuration options for the duration, number of failures necessary to trigger a ban, etc. By combining fail2ban with Limit Login Attempts, we can leverage both a WordPress-level solution to identify problem users easily and a kernel firewall-level solution to remove them from the server with minimal impact.

Fail2ban operates on log file parsing, which means it is incompatible with currently released versions of Limit Login Attempts. I will talk to the author and see if I can get him to incorporate writing to an actual log file in addition to logging in the MySQL database running wordpress, but for now, we’ll simply hack in the functionality we need, because it only takes a few lines. In limit-login-attempts.php, in the function limit_login_notify_log, which starts around line 563, we’ll append the following lines to the end of the function, making it write a simple message with the user’s IP address to a pre-specified and hardcoded log file name whenever a lockout is also written to the database:

$file = fopen(LIMIT_LOGIN_FILE_PATH, 'at');
if ($file) {
// Note that the 't' flag will offer \n to \r\n translation on windows
fwrite($file, date('M j h:i:s ') . sprintf(__('IP locked out by wp-limit-login: %s', 'limit-login-attempts'), $ip) . "\n");
fclose($file);
}

Then, we also need to add a line defining the constant LIMIT_LOGIN_FILE_PATH, so I’ve added define(LIMIT_LOGIN_FILE_PATH, ‘/var/log/nginx/wp-limit-login.log’); at the top of the file. Perfect, now whenever logging is enabled (and why would you disable it?), the offending user’s information will be written to both the MySQL database to display in the Limit Login Attempts dashboard and to the specified log file. Make sure the file path you choose for the log file is writeable by whichever process runs your php instance (be it apache, php-fpm, etc). This log file is very simple, which suits my purposes, but you may want to augment yours with some other information, such as potentially the name of the blog, if you’re running a multi-site installation, or the number of times an IP has been locked out. It’s also entirely possible to put this file inside your wordpress install, if you don’t have access to /var/log for instance. If you put it inside the plugin directory itself, the standard wordpress rewrite rules should prevent clients from accessing it, but I haven’t verified this, so I’d recommend keeping it outside of your webroot to avoid disclosing the IP addresses (we’re responsible administrators, even if the people we’re banning aren’t).

Next, we add a filter to the fail2ban configuration. Filters are defined in /etc/fail2ban/filter.d, one per file. I suggest a simple name, like wplogin.conf. Inside this filter, we configure the regular expression to capture a failure and an optional regular expression to ignore, which trumps a failure line. Based on the output above, our filter is very simple:

# Capture failed login attempts through wordpress limit logins module

[Definition]

# default message is "IP locked out by wp-limit-login: <ip address>"
failregex = [^:]*: <HOST>

ignoreregex =

With the filter in place, we must add a jail configuration to fail2ban. For that, we edit /etc/fail2ban/jail.conf and add a new jail that references the filter we just created. I would strongly recommend that you add your own IP address on the ignoreip line , in CIDR notation, because if not, all of your traffic to your server will be dropped (for a specified amount of time), which will make it impossible for you to access.

[wplogin]
enabled = true
filter = wplogin
action = iptables-allports[name=wplogin]
logpath = /var/log/nginx/wp-limit-login.log
maxretry = 1
bantime = 600
ignoreip = 127.0.0.1/32

Because Limit Login Attempts already counts how many times someone has failed to log in, an entry here means they should be banned. Therefore, we set the retry count to just one, so that it will ban them on their first attempt. We also use the iptables-allports action to ban the user’s IP address, which does almost exactly the same thing as the script I presented before, except that it uses a separate fail2ban chain and names the rules so that it can go back and remove them later with minimal fuss (i.e. it is more sophisticated). I would also recommend increasing the ban time, as the default is only 10 minutes. Unless you run into problems with actual readers being banned (unlikely, as the botnet appears to run on other servers, not on infected clients), you can easily set this to an hour, or a day, at your discretion.

After that, simple launch fail2ban by executing:

# fail2ban-client start

and fail2ban will begin monitoring the log file. You may notice that fail2ban gives a warning on startup saying that the log file doesn’t exist. If this happens, create the file, change its owner to match the account that will be writing to it, and restart fail2ban:

# touch /var/log/nginx/wp-limit-login.log
# chown nginx /var/log/nginx/wp-limit-login.log
# fail2ban-client reload

This should combine the power of Limit Login Attempts to protect your installation from brute force password attacks and fail2ban to drop traffic to your server, improving its effectiveness. The combination will reduce the amount of server resources spent dealing with illegitimate requests, which can turn an almost-DoS caused by brute force password hammering into a non-issue.

Apr 072013
 

While doing some updates to the blog, such as adding author information and other useful information to hopefully help with legitimatizing things, I spent about two hours trying to figure out why a shortcode in the theme that I use (Suffusion) would not parse arguments. The tag suffusion-the-author would display author name, but if I added an argument to make it display the description or a link instead of just the name, it would always fail and revert to the author name only. Some quick googling told me that this didn’t seem to be a problem anyone else had, and after four major versions for the theme, I’m pretty sure someone would’ve noticed if a basic feature were entirely nonfunctional. After lots of debugging, I discovered that the problem was happening in WordPress’s code, inside the definition of shortcode_parse_atts(), which parses the attributes matched in the shortcode into an array for use inside the callback. For me, instead, it killed all of the tag arguments and returned an empty string. In the end I found that the issue was a regular expression that appears to be designed to replace certain types of unicode spaces with normal spaces to simplify the argument parsing regular expression that comes after

$text = preg_replace("/[\x{00a0}\x{200b}]+/u", " ", $text);

On my system, this would simply set $text to an empty string, every time. Initially, I wanted to just comment it out, because I’m not expecting to see any of these characters inside tags, but that bothered me, because this has been a part of the WordPress code for a very long time, and nobody else appears to have problems with it. Finally, I concluded that this must mean that unicode support was not enabled, so I began searching through yum. Bad news, mbstring and pcre were all already installed, and php -i told me that php was built with multibyte string support enabled, as well as pcre regex support. I tested my pcre libraries and they claimed to support unicode as well.

In the end, I solved the problem by updating php and pcre to the latest version, which was an update from php-5.3.13 to php-5.3.24 in my case, and pcre was updated from 7.x to 8.21. It appears that there was some kind of incompatibility between php 5.3 and the 7.x branch of pcre (if you build php 5.3 from source, it will target pcre 8.x), which prevented me from using unicode despite support for it being present everywhere. So, if you have trouble getting your php installation to handle unicode inside regular expressions, and you’re running on Amazon Web Services, make sure you update to the latest versions of php and pcre, as there appear to be some issues with older packages.

Apr 022013
 

Yesterday, I had a small issue where someone attempted to brute force the admin password to my blog, resulting in significantly decreased availability for about 10 minutes as my server’s resources were maxed out, leading to 45+ second page load times. Luckily, it happened just as I was coming home, so I was able to identify the problem and put a stop to it very quickly. Incidentally, this taught me that wordpress does not have a rate limiting algorithm in place on logins (allowing brute force attacks at the server’s maximum request speed). So, I would recommend using a plugin to add rate limiting to logins, which can mitigate the danger associated with a simple scripted attack.

Once the problem was identified as an attack of some kind, the next step was to disable php-fpm in order to drop the load average to an acceptable level so that I could figure out who and what was going on, and then do my best to mitigate it. Incidentally, this will cause nginx to produce a 502 error as the CGI handler is no longer responsive. From /var/log/nginx/blog.access.log:

{IP} - - [01/Apr/2013:16:59:57 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:57 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:58 +0000] "POST /wp-login.php HTTP/1.1" 200 6969 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-admin/ HTTP/1.1" 302 5 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:16:59:59 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2Fthelonepole.com%2Fwp-admin%2F&reauth=1 HTTP/1.1" 200 6044 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

becomes

{IP} - - [01/Apr/2013:22:45:53 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "GET /wp-admin/ HTTP/1.1" 502 173 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
{IP} - - [01/Apr/2013:22:45:54 +0000] "POST /wp-login.php HTTP/1.1" 502 198 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Interestingly, we can see that whatever brute force attack is being used attempts to spoof the bing search crawling robot in order to prevent me from using a user-agent denial without deindexing myself. This isn’t really a big deal, because I don’t plan on solving the problem with policies in nginx, so anything in the http request is largely irrelevant.

I’ve removed the user’s IP address, but in this case it was assigned through a cloud services provider, meaning that someone likely set up an account to look for wordpress blogs on the internet and attempt to break into their admin directory, likely to install a plugin and then take over the machine in order to attach it to a botnet. I’ve emailed their abuse department, but I never heard back and don’t really expect them to do anything about it (most companies don’t seem to care). I don’t expect any of my traffic to come from cloud providers, so the simplest solution is to deny all traffic from the IP in question. There are a number of ways for me to do this, but the best is to drop requests as early as possible in my network stack, because it minimizes the amount of resources that my server will dedicate to processing an illegitimate request. To do this, I use the kernel firewall utility iptables, adding rules to drop all incoming packets (with no acknowledgement at all, which makes it appear as though my server is no longer online) from blacklisted sources as early as possible in the processing chain.

General scanning of the internet, probing for vulnerabilities, and attempting to exploit poorly configured servers is extremely common. I used to get very concerned when I saw logfile entries almost daily indicating that someone had attempted to request files that don’t exist or exploit a flaw in how older versions of apache (which I don’t use) looks up files. After doing some reading, I relaxed, because it’s very common, and servers running up-to-date software will withstand generic attacks (but likely not actual, targeted attempts at breaching). That said, since I have no interest in letting these people continue to scan me for vulnerabilities in order to make me a part of their botnet, and they almost certainly do not form any part of my readership, I put together a small script that added a log message and dropped the packets whenever an IP I disliked attempted to reach the server, to make it quick and easy to deny people whose nonsense was becoming problematic.

#!/bin/bash

if [ -z $1 ]; then
        echo "Usage: $0 <ip address> [reason]";
        exit;
fi

IP=$1
REASON="denied ip"

if [ $# -gt 1 ]; then
        shift;
        REASON=$@
fi

iptables -A INPUT -s ${IP}/32 -j LOG --log-prefix "${REASON} "
iptables -A INPUT -s ${IP}/32 -j DROP

You can see log messages (including those generated by iptables) in /var/log/messages where you will see a message for each dropped packet. This could potentially cause your log file to balloon, in the case of any kind of flooding, so you may wish to eliminate the log rules, but I generally keep them so that I remember later why an IP address was blocked.

Feel free to use this whenever you find you need to deny access to a specific IP address, to save the trouble of learning iptables’ options. If you add lots of entries to your INPUT chain in iptables, all of your network processing will slow down, as each incoming packet is matched against all of the rules in the chain in order until it reaches the end, where it is passed up to the next layer. That said, the slowdown is relatively minor and does not become noticeable until you have added hundreds of rules. If you find yourself in this situation, perhaps investigate another approach, or consider using multiple chains and branching as early as possible, rather than listing all blacklisted IPs sequentially, to limit the maximum chain traversal length.

Dec 182012
 

For a project that I started working on with a friend back in November, we needed to use a project management system for bugs and progress tracking, and we decided to try Trac an open project management system with svn integration written in python.

I run nginx to do all of my web serving needs, so I followed the instructions given for nginx configuration for Trac, and I was a little disappointed to find that the FastCGI script given didn’t do any normal server things, like fork a child process to run in the background. I also wanted to integrate the FastCGI server with the service management tools available in CentOS, so I set to work.

First, I modified the nginx-specific FastCGI script to accept a number of command line arguments, allowing me to use the same script to run multiple track servers for multiple different projects. In total, it accepts a project path, a user, a unix socket to listen for fcgi connections, and a location to store the server process pid. Then, I added the child fork to allow me to run it as a service. Here is the version that I use now:

#!/usr/bin/env python
from optparse import OptionParser
import os
import pwd

parser = OptionParser()
parser.add_option("-s", "--sock", dest="sockaddr", help="Use a specific unix socket", default="/var/trac/run/instance.sock")
parser.add_option("-u", "--user", dest="user", help="Run FCGI process as a specific user", default="nginx")
parser.add_option("-t", "--trac", dest="trac_env", help="Trac environment path to use", default="/var/trac/project")
parser.add_option("-p", "--pid", dest="pidfile", help="Location to store pid file", default="/var/run/trac.fcgi.pid")
(options, args) = parser.parse_args()

os.environ['TRAC_ENV'] = options.trac_env

try:
     from trac.web.main import dispatch_request
     import trac.web._fcgi

     pid = os.fork()
     if pid == 0:
          os.setuid(pwd.getpwnam(options.user)[2])
          fcgiserv = trac.web._fcgi.WSGIServer(dispatch_request, bindAddress = options.sockaddr, umask = 7)
          fcgiserv.run()
     else:
          os.system("echo " + str(pid) + " > " + options.pidfile)

except OSError, oe:
    raise
except SystemExit:
    raise
except Exception, e:
    print 'Content-Type: text/plain\r\n\r\n',
    print 'Oops...'
    print
    print 'Trac detected an internal error:'
    print
    print e
    print
    import traceback
    import StringIO
    tb = StringIO.StringIO()
    traceback.print_exc(file=tb)
    print tb.getvalue()

With a service-capable FastCGI script, now I needed to create an /etc/init.d script to control the process. Because I am running CentOS, I also wanted the script to be compatible with chkconfig so that it can be used with the service utility. This was pretty simple, as I just copied an existing init script and ripped out most of its guts, replacing with a minimal set of controls: start, stop, and restart. Here’s the full init script, which I saved as /etc/init.d/trac-fcgi :

#!/bin/sh
#
# chkconfig:   - 85 15
# description:  Runs trac's fcgi script as a daemon
# processname:  trac.fcgi
# pidfile:     /var/run/trac.fcgi.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0

tracfcgi="/var/trac/trac.fcgi"
prog=$(basename $tracfcgi)
pidfile="/var/run/${prog}.pid"

start() {
    echo -n $"Starting $prog: "
    daemon $tracfcgi
    retval=$?
    echo
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc -p $pidfile $prog
    retval=$?
    echo
    return $retval
}

restart() {
	stop
	start
}

rh_status() {
	status $prog
}

rh_status_q() {
    rh_status >/dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart)
        $1
        ;;
    status|status_q)
        rh_$1
        ;;
    condrestart|try-restart)
        rh_status_q || exit 7
        restart
	    ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart}"
        exit 2
esac

Finally, the new init script is added with a simple call to chkconfig:

# chkconfig --add trac-fcgi

If you want to run multiple trac servers with different configurations, you will need to create a copy of the init.d script per configuration and add them using command line arguments on the line that reads daemon $tracfcgi after the $tracfcgi variable. Then, add each configuration’s script to chkconfig using the same method as before.

May 102012
 

A few weeks ago, the online gaming community that I belong to (The Legion) deployed a new website to integrate information about the professional e-sports teams with the existing community, forum-based website. As a part of that, we wanted to include the ability to embed and view streams on the home page, with additional information about stream status. Luckily, twitch.tv includes a very powerful and flexible REST API for determining if a stream is online, with output formats in XML, JSON, or JSONP. After the preliminary support for twitch.tv was added, which covered most of the community, we wanted to add the ability to embed streams from another popular streaming website, own3d.tv, as there are a few popular players on the team who use it instead.

Unfortunately, own3d.tv’s API is significantly lacking in features. While twitch.tv returns extensive metadata about the channel, including game played, stream title, streamer name (popular channels can have multiple people stream to them at different times), viewers, preview image URLs, and other interesting information, the entire own3d.tv response consists of stream status (live/not), viewer count, and time spent streaming. To make matters worse, the own3d.tv API is undocumented and appears to have a single endpoint that returns data only in XML format: http://api.own3d.tv/liveCheck.php?live_id=(integer id here). The twitch.tv REST API allows a user to request information about any number of channels simultaneously, while the own3d API appears to only allow a single query per request.

Due to the way that cross-domain AJAX requests are handled (i.e. they aren’t), JSONP emerges as the most useful format for a REST API to be used in building a web application using javascript. I don’t know if this is the standard approach, but the easiest thing to do with twitch.tv appears to be to define a handler function, such as handleTwitchResponse and then insert a script tag into the HTML referencing the REST API endpoint as a source for script, with output set to JSONP and the appropriate handler supplied:

<script type="text/javascript" src="http://api.justin.tv/api/stream/list.json?channel=channel1,channel2,channel3&jsonp=handleTwitchResponse">

The resulting object supplied by twitch.tv’s servers will be wrapped in a function call to handleTwitchResponse, which immediately hooks it into your own local processing system. It would be great if own3d.tv supported the same approach (for flexibility), but it does not, so we are forced to create our own workaround.

To that end, I wrote a short php script that exposes an interface similar to the one used by twitch.tv, where a JSONP handler can be supplied, along with multiple stream IDs, and the results will be returned in an array:

/**
 * Script to forward requests to own3d.tv and return them in JSONP format for our
 * local stuff to handle.
 */

$ids = explode(',', $_GET['live_id']);
$outout = array();

foreach ($ids as $id) {
	$xml = file_get_contents('http://api.own3d.tv/liveCheck.php?live_id='.((int)$id));
	$sxml = new SimpleXMLElement($xml);
	$results = array('isLive' => (string)$sxml->liveEvent->isLive,
					'liveViewers' => (int)$sxml->liveEvent->liveViewers,
					'live_id' => (int)$id,
					'liveDuration' => (int) $sxml->liveEvent->liveDuration);
	$output[] = $results;
}

if (isset($_GET['jsonp']))
	echo $_GET['jsonp'].'('.json_encode($output).');';
else
	echo json_encode($output);

The script itself is pretty simple, as mostly it just glues together a few pieces and changes the format of the data. A comma-separated list of IDs is expected as a GET parameter, along with an optional JSONP callback, presenting a format almost identical to the twitch.tv interface. If no callback is supplied, standard JSON formatting is used instead. Using the ability to make HTTP requests via fopen() (which requires Allow fopen URL wrappers to be set to true in php.ini), it fetches information for each ID individually from own3d.tv, then parses the XML into a PHP array, and spits it out using the built-in JSON-encoding functions provided within PHP.

Note: This script does not translate the names used by own3d.tv to match those used by twitch.tv, but it does add the live_id to the response, in order to allow you to identify which stream a particular status object corresponds to when receiving multiple aggregated into a single response. I did this somewhat intentionally, because it is more useful as a drop in replacement for systems that wish to conserve the own3d.tv naming conventions. You can also readily rename the javascript object properties after all the data has been received by the client, before forwarding the object to a standard twitch.tv handler, which means that it’s also kind of a moot argument.

The script should be stored locally on your server. Then, it can be accessed with standard AJAX, or, for consistency, in a manner identical to how the twitch.tv script is referenced:

<script type="text/javascript" src="own3d_check.php?live_id=1234,1235,1236&jsonp=handleOwnedResponse"></script>

If you find yourself in a situation where you need to deal with own3d.tv streams and their API before it is improved, feel free to use the above script as a reference (or just borrow it I suppose?). Hopefully they will realize that twitch.tv is leagues ahead of them at the moment in this aspect of web streaming, and in order to stay competitive they need to address API needs of external websites.