While discussing game scripting languages with a friend, I asserted that a Lisp-based language such as Scheme or Racket would form a very effective in-game scripting language due to its expressive power and flexibility, as well as the efficiency with which modern, JIT-capable implementations execute code. One of the most widely used scripting languages for this purpose appears to be Lua which is pretty small and designed to be embedded in C programs. I glanced over the feature list on Wikipedia and saw that it was basically a subset of Scheme with some omissions and a few additions. In particular, I really liked the idea of a metatable, which effectively automatically handled function memoization.

The example given on Wikipedia is a metatable-based function to calculate the nth Fibonacci number, a classic example of a simple recursive expression that has exponential time complexity for a naïve implementation, but can be improved to linear time (or, depending on the usage scenario, amortized constant time, for sufficient reuse) with a memoization table. The Lua for this is given (available under the Creative Commons Attribution-ShareAlike license from Wikipedia as:

fibs = { 1, 1 }
setmetatable(fibs, {
  __index = function(name, n)
    name[n] = name[n - 1] + name[n - 2]
    return name[n]
  end
})

I didn't recall seeing a Scheme or Racket built-in to provide the same functionality, but I was confident that it would be easy to implement such a language feature using standard Racket built-ins, such as the hash table. So, I started with a simple function that would check if a value for 'arg' existed in a given hash table, and if so, return that value, otherwise compute a value by applying the given function, storing the result, and then returning:

(define (memo-or-call ht fun arg)
  (if (hash-has-key? ht arg)
      (hash-ref ht arg)
      (let ((result (fun arg)))
        (hash-set! ht arg result)
        result)))

With this, I was able to define a hash table on my own, set up the default values, and then create an index function, which was pretty effective and enabled me to compute any Fibonacci number that I wanted almost instantly:

(define fibtable (make-hash))
(hash-set! fibtable 0 1)
(hash-set! fibtable 1 1)

(define (fib n)
  (memo-or-call fibtable
                (lambda (k) (+ (fib (- k 1)) (fib (- k 2))))
                n))

This is great, but it felt like it required too much template-like rewriting of code for each function that I wanted to provide table memoization for. I'd have to create a new hash table, add default values, and then use the "internal" function memo-or-call to create my memoization-capable function. To resolve this, I wrote another function that would take a lambda expression and wrap it with a hash table and call to the memoization function:

(define (mt-store-defaults tbl defaults)
  (cond
    [(empty? defaults) tbl]
    [else (hash-set! tbl (caar defaults) (cadar defaults))
          (mt-store-defaults tbl (cdr defaults))]))

(define (make-metatable fun defaults)
  (let ((tbl (mt-store-defaults (make-hash) defaults)))
    (lambda (k)
      (memo-or-call tbl fun k))))

This reduced the complexity of creating a new memorized function considerably, which brought me to functionality that was effectively equivalent to what was possible in Lua. It's worth noting that I'm strictly dealing with the case where a table is used to memorize computation; not the other usage cases, such as implementing classes or objects, although it would be possible to modify the idea here to include support for the __call magic function in Lua, and it is already possible to store functions in a hash table, as they are still first class objects. Scheme code implementing the now-table-based Fibonacci calculator

(define fib2
  (make-metatable
   (lambda (k) (+ (fib2 (- k 1)) (fib2 (- k 2))))
   '((0 1) (1 1))))

Therefore, in summary, a reasonable implementation of automatic memoization with "metatables" can be achieved with just a few lines of Scheme/Racket, allowing for concise definitions of memorized functions in user code. Note that this is just some example code and not really robust, but feel free to use it as a starting point.

#lang racket

(define (memo-or-call ht fun arg)
  (if (hash-has-key? ht arg)
      (hash-ref ht arg)
      (let ((result (fun arg)))
        (hash-set! ht arg result)
        result)))

(define (mt-store-defaults tbl defaults)
  (cond
    [(empty? defaults) tbl]
    [else (hash-set! tbl (caar defaults) (cadar defaults))
          (mt-store-defaults tbl (cdr defaults))]))

(define (make-metatable fun defaults)
  (let ((tbl (mt-store-defaults (make-hash) defaults)))
    (lambda (k)
      (memo-or-call tbl fun k))))

Earlier this week, I stumbled across McCarthy's original paper describing LISP, and I felt a motivation to write my own LISP interpreter. I know there are plenty of good ones out there already, but this isn't really about making one that is better or different from the others. Mostly, I am interested in exploring my own understanding of what LISP programs really mean and, as a part of that, testing my ability as a C programmer to implement a LISP interpreter. I've talked with a friend that I will be collaborating with on this project, and we hope to get a basic interpreter up and running over the weekend.

We don't have a definitive feature list, although we do plan on supporting everything in the original paper, as well as basic garbage collection and an external function interface to define symbols that provide additional functionality to the LISP environment. This is primarily an investigation into what we're able to do, and, as a part of that, how hard it is to write a functional LISP interpreter, so we're not planning on too many language features; the focus is mostly on things that are necessary to run the engine itself. If we wanted the kind of functionality found in PLT Scheme/Racket, then we'd just use that to run our programs, but we don't want to stop with something that can't be used to do anything useful, either.

We've thrown around the idea of making a stand-alone compiler as well, but for now we're simply going to focus on implementing a REPL and on executing LISP scripts as an interpreter. I know that technically this counts as compilation (in some sense, based on the semantics of the paper), but we've also considered making a standalone compiler that will take a script and produce a binary that does not need to re-parse the script every time it is executed.

If you want to follow our progress, we've created a public github repository: https://github.com/gmalysa/cs-lisp where you can see all of the code.

I normally don't write about videogames, but I thought it would be fun to take some time to write about the armor and resistance system in Diablo III, as well as to throw some math at it to see how best to improve one's character.

In Diablo III, the key to surviving Inferno is getting appropriate gear. However, aside from the obvious improvements of increasing the stats on an item, it can be difficult to intuitively judge the tradeoffs between dissimilar stats. Is it worth it to give up 40 vitality points in order to gain 20 points of resistance to all? How about trading 100 points of strength for 100 points of intelligence? In trying to answer these questions, we will temporarily ignore the question of damage, because, if you can't survive an attack from an Inferno-level elite, unless you do over a million DPS, you won't have a chance to kill it anyway. There are two basic concepts to understanding survivability when absorbing damage (we will ignore dodge because it further complicates things): damage reduction and effective hit points, which are really two ways of looking at the same stats.

Damage Reduction

The combat system is pretty straightforward: when a monster attacks, a random number is generated between its min and max attack values, that number is reduced by armor, resistance, and extra modifiers, and then the resulting value is subtracted from your health. The consensus (which can be experimentally verified) is that the resistance and armor reductions can be found with the following formulas:

r_{resist}=\frac{R_{all}}{5m+R_{all}}, r_{armor}=\frac{A}{50m+A}

Where R represents resistance total, A represents armor total, m represents the monster level (60 for all of inferno, as far as I can tell), and r is the reduction, which is expressed on the UI as a percentage. Therefore 1-r is used to calculate how much damage is done by an attack. The two resistances multiply, so the total damage done by a single attack is given as $ (1-r{resist})(1-r{armor})$ multiplied by the initial attack damage. Note that if you find a shrine of protection or use a skill such as the wizard's blur ability, it is counted in a third category that we will ignore from here out. This category is multiplied in exactly the same way, so total damage becomes (1-r_{resist})(1-r_{armor})(1-r_{other}) in that case.

The reduction is the remainder after the damage amount is subtracted from the total damage. If we normalize by the monster's damage, we find that the damage reduction for any attack becomes:

DR=1-(1-r_{resist})(1-r_{armor})

To increase one's survivability, it is necessary to increase the value of damage reduction, but because stats are limited on items, there is always a tradeoff. Because there is a direct, unchanging relationship between strength an armor (1 strength adds 1 armor) and between intelligence and resistance to all (10 intelligence adds 1 to all resistance), we will first look at the tradeoff between armor and resistance alone. A quick look at the two equations tells us that if we multiply r_{resist} by 10/10, we will get the same expression that is used to find the reduction by armor, which suggests that 1 resistance to all is equal to 10 armor because they increase their respective reductions at the same rate. This is only accurate in certain cases, because the total damage reduction depends on the product of all resistance and armor, rather than a linear combination of the two.

Instead, it is much more useful to examine the partial derivatives of damage reduction with respect to all resistance and armor, respectively:

\frac{\partial {DR}}{\partial A}=(1-r_{resist})(\frac{\partial }{\partial A}\frac{A}{50m+A})

=(1-r_{resist})(\frac{1}{50m+A}-\frac{A}{(50m+A)^2})=(1-r_{resist})(\frac{50m}{(50m+A)^2})

The partial derivative of damage reduction due to all resistance can be found analogously to be:

\frac{\partial {DR}}{\partial R_{all}}=(1-r_{armor})(\frac{5m}{(5m+R_{all})^2})

The relationship between all resistance and armor is, therefore, dependent on what the two values are, and we can see that increasing the smaller value will have a larger impact on total damage reduction than further increasing the larger value will, which means that 1 all resistance is not always worth the same as 10 armor, when it comes to changing damage reduction—sometimes all resistance is more valuable, and sometimes less, with the 1:10 ratio being equal when  r_{resist} and r_{armor} are equal. This is kind of hard to visualize, so it's much more useful to simply create a contour plot of damage reduction as a function of both armor and all resistance, which serves as a guide for the relationship between the two stats.

Damage Resistance Contour Plot

Moving to a darker red area represents an increase in total damage reduction, and so we see that along the diagonal, both contribute equally, but when one variable is significantly larger than the other, further increases are not as effective as bumping the lower stat would be.

Effective HP

With a good understanding of how damage resistance is calculated, it is time to extend the concept to allow us to include other tradeoffs when determining which gear to use. The next component of survivability is how many hit points your character has. However, all characters with 30,000 HP are not created equal—the character with better damage reduction will do better in combat, so our goal is to quantify the relationship between hit points and damage reduction.

In order to compare characters with different resistances and HP, it is necessary to eliminate the armor and resistance entirely and focus on only a single point of comparison that summarizes the effects of all of the attributes, which is known as 'Effective HP.' If a character with 90% damage reduction is hit and suffers 10,000 HP worth of damage, by inverting the equation we can find that the attack must have done 100,000 HP of damage originally, and so, an equivalent character with zero armor and zero resist would have an effective 100k HP for each 10k HP that our character has. The direct formula for effective HP from actual HP and damage reduction is:

HP_e=\frac{HP}{1-DR}

At this point, more derivatives could be calculated to find the effects of changing vitality, armor, or resistance on effective HP, but it is much simpler to create a spreadsheet that computes the effective HP for a given triple of vitality (plus your character's base hit points, which is not zero), armor, and all resistance, and then using that to compare different gear setups. Intuitively, more effective HP is better, as it means you will survive longer against elites and champions, regardless of whether your actual HP increased or decreased. The same is true for your damage reduction; your effective HP can increase despite damage reduction decreasing, if your base HP is sufficiently increased at the same time. It is also worth noting that all HP regeneration becomes more effective as damage reduction goes up—the effective HP regeneration can be calculated in exactly the same way as effective HP is calculated.

Hopefully it is clear that selecting Diablo III gear is a constrained optimization problem. Maybe one day I'll take a stab at writing a program to optimize stats or provide some kind of discussion on the actual tradeoffs between DPS and HP as they relate to killing monsters, but that sounds like a very painful exercise when approximate solutions can be found just by thinking +effective HP and +damage are good.

2018 Note: justin.tv no longer exists, own3d.tv no longer exists, and twitch.tv uses a different API, so most of the information here is only really interesting to see how this was done historically, but can't be used to build a similar feature currently. The documentation links might not even work.

A few weeks ago, an online gaming community that I belong to (The Legion) deployed a new website to integrate information about the professional e-sports teams with the existing community, forum-based website. As a part of that, we wanted to include the ability to embed and view streams on the home page, with additional information about stream status. Luckily, twitch.tv includes a very powerful and flexible REST API for determining if a stream is online, with output formats in XML, JSON, or JSONP. After the preliminary support for twitch.tv was added, which covered most of the community, we wanted to add the ability to embed streams from another popular streaming website, own3d.tv, as there are a few popular players on the team who use it instead.

Unfortunately, own3d.tv's API is significantly lacking in features. While twitch.tv returns extensive metadata about the channel, including game played, stream title, streamer name (popular channels can have multiple people stream to them at different times), viewers, preview image URLs, and other interesting information, the entire own3d.tv response consists of stream status (live/not), viewer count, and time spent streaming. To make matters worse, the own3d.tv API is undocumented and appears to have a single endpoint that returns data only in XML format: http://api.own3d.tv/liveCheck.php?live_id=(integer id here). The twitch.tv REST API allows a user to request information about any number of channels simultaneously, while the own3d API appears to only allow a single query per request.

Due to the way that cross-domain AJAX requests are handled (i.e. they aren't), JSONP emerges as the most useful format for a REST API to be used in building a web application using javascript. I don't know if this is the standard approach, but the easiest thing to do with twitch.tv appears to be to define a handler function, such as handleTwitchResponse and then insert a script tag into the HTML referencing the REST API endpoint as a source for script, with output set to JSONP and the appropriate handler supplied:

<script type="text/javascript"
src="http://api.justin.tv/api/stream/list.json?channel=channel1,channel2,channel3&jsonp=handleTwitchResponse">
</script>

The resulting object supplied by twitch.tv's servers will be wrapped in a function call to handleTwitchResponse, which immediately hooks it into your own local processing system. It would be great if own3d.tv supported the same approach (for flexibility), but it does not, so we are forced to create our own workaround.

To that end, I wrote a short php script that exposes an interface similar to the one used by twitch.tv, where a JSONP handler can be supplied, along with multiple stream IDs, and the results will be returned in an array:

/**
 * Script to forward requests to own3d.tv and return them in JSONP format for our
 * local stuff to handle.
 */

$ids = explode(',', $_GET['live_id']);
$outout = array();

foreach ($ids as $id) {
    $xml = file_get_contents('http://api.own3d.tv/liveCheck.php?live_id='.((int)$id));
    $sxml = new SimpleXMLElement($xml);
    $results = array('isLive' => (string)$sxml->liveEvent->isLive,
                    'liveViewers' => (int)$sxml->liveEvent->liveViewers,
                    'live_id' => (int)$id,
                    'liveDuration' => (int) $sxml->liveEvent->liveDuration);
    $output[] = $results;
}

if (isset($_GET['jsonp']))
    echo $_GET['jsonp'].'('.json_encode($output).');';
else
    echo json_encode($output);

The script itself is pretty simple, as mostly it just glues together a few pieces and changes the format of the data. A comma-separated list of IDs is expected as a GET parameter, along with an optional JSONP callback, presenting a format almost identical to the twitch.tv interface. If no callback is supplied, standard JSON formatting is used instead. Using the ability to make HTTP requests via fopen() (which requires Allow fopen URL wrappers to be set to true in php.ini), it fetches information for each ID individually from own3d.tv, then parses the XML into a PHP array, and spits it out using the built-in JSON-encoding functions provided within PHP.

Note: This script does not translate the names used by own3d.tv to match those used by twitch.tv, but it does add the live_id to the response, in order to allow you to identify which stream a particular status object corresponds to when receiving multiple aggregated into a single response. I did this somewhat intentionally, because it is more useful as a drop in replacement for systems that wish to conserve the own3d.tv naming conventions. You can also readily rename the javascript object properties after all the data has been received by the client, before forwarding the object to a standard twitch.tv handler, which means that it's also kind of a moot argument.

The script should be stored locally on your server. Then, it can be accessed with standard AJAX, or, for consistency, in a manner identical to how the twitch.tv script is referenced:

<script type="text/javascript" src="own3d_check.php?live_id=1234,1235,1236&jsonp=handleOwnedResponse">
</script>

If you find yourself in a situation where you need to deal with own3d.tv streams and their API before it is improved, feel free to use the above script as a reference (or just borrow it I suppose?). Hopefully they will realize that twitch.tv is leagues ahead of them at the moment in this aspect of web streaming, and in order to stay competitive they need to address API needs of external websites.

A few days ago, I had what I expect is a common problem: I needed to print off some PDFs, sign them, and then scan them back into PDF format to return. However, I didn't have any software for scanning directly to PDF, and most solutions on the internet appear to be either sketchy or cost money. The scanner functionality that is provided with Windows for working with TWAIN devices can create TIFF format images of the scanner output, so all I really needed was a solution that would allow me to create a PDF and insert a TIFF image. Luckily, there is a PECL package for creating PDF files using a PHP script, so I wrote a small command line PHP script to take a series of TIFF images and create a PDF, with one image per page.

Setting up PHP's PDFLib Front End

Because the front-end for PDFLib is (now) a PECL package, it is not bundled with PHP, so you must retrieve it using the PEAR tool. Unfortunately, the PHP frontend requires a separate installation of PDFLib, so I began by obtaining a copy of the free for non-commercial use of PDFLib-Lite. As a home user only interested in converting some things that I scanned to email to a friend, I believe that I qualify for non-commercial use. Because there is no RPM of PDFLib Lite for Amazon's Basic Linux AMI, I had to build from source, which was routine except that the initial image does not include gcc for some reason, so I had to install it as well. For reference, or for those new to Linux:

# wget http://www.pdflib.com/binaries/PDFlib/705/PDFlib-Lite-7.0.5.tar.gz
# tar -zxvf PDFlib-Lite-7.0.5
# cd PDFlib-Lite-7.0.5
# ./configure
# make
# make install

Now that PDFlib-Lite is installed to /usr/local it is necessary to download and install the PECL extension to expose it to PHP. On an Amazon virtual server instance it is necessary to install the php-devel package as well as the php-pear package first with yum, in order to have the necessary source and headers to build extensions, along with the tools to access PEAR; others may need to install it as well, depending on their setup. After that, installing new PHP extensions via PEAR is easy:

# pear install pecl/pdflib

Finally, add the extension to your configuration file, /etc/php.ini. Some configurations recommend putting extensions in their own files, which I think is kind of silly, but I'll play along, so an alternative is to create a file in /etc/php.d/ named pdf.ini with only one line:

extension=pdf.so

In the event that after installing pdflib your extension has a different name, PEAR will tell you what to use after it is done installing.

Using PHP to Create PDFs

With all the libraries installed, I just needed a simple script that would create a new PDF, load up some TIFF-format images, and then insert them as full pages. So, I wrote this small command line script that accepts a list of input files and produces a PDF using one image per page, with each image resized to occupy the entire 8.5"x11" page. It can be run in Windows or Linux using the PHP CLI, given that the appropriate software is installed, because it should be invoked with the php -f pdf_convert.php [arguments ...] syntax from any prompt, rather than relying on a #!/bin/php at the top of the file.

Please see this gist for the source code, as the listing below may contain errors due to markup issues (in particular PHP code sometimes triggers LaTeX processing of statements between two dollar signs as a math expression, even inside a code block)

<?php
/**
 * Convert a series of TIFF files given on the command line to pages in
 * a single PDF
 */

if ($argc < 3) {
    echo ('Convert a series of TIFF images as individual pages in a PDF');
    echo ("\n".'Usage: php -f pdf_convert.php <output name> <input 1> <input 2> ...');
    exit(0);
}

// Get file names from command line
$output_file = $argv[1];
$input_files = array();
for ($i = 2; $i < $argc; ++$i) {
    $input_files[] = $argv[$i];
}

// Initialize PDF File
$pdf = new PDFLib();
if ($pdf->begin_document('', '') == 0) {
    die('Error: '.PDF_get_errmsg($pdf));
}

// For each of the input images, load them, create an 8.5" x 11" page, place them as
// the full page, and then close the page
foreach ($input_files as $in_file) {
    echo ('Now processing '.$in_file.'...'."\n");
    $pdf->begin_page_ext(612, 792, '');
    $image = $pdf->load_image('auto', $in_file, '');
    $pdf->fit_image($image, 0, 0, 'boxsize {612 792} fitmethod meet');
    $pdf->close_image($image);
    $pdf->end_page_ext('');
}

$pdf->end_document('');

$raw_pdf = $pdf->get_buffer();

file_put_contents($output_file, $raw_pdf);

echo ('Done!'."\n");

This doesn't do much in the way of error checking or proper handling, because it didn't seem necessary for something I'm only going to use once or twice and isn't really a packaged product. Feel free to use this or modify it for whatever you like, under the terms of the GPL.