colinrmitchell.com

Blog

Skulltag Server Log File Analysis Scripts

Posted Tuesday, April 5th 2011 in Programming - Permalink

I have been working on some scripts lately that let me see who has been visiting my Doom servers. The latest version of these scripts can be obtained via SVN from here. The latest version that I get around to packaging can be found here.

The scripts are written in Python, focusing on *nix systems, and produce HTML pages to view the data gleaned from the log files. The first script chugs through the raw log file and grabs player names, IP addresses, and connection/disconnection times. From this, three output data files are produced, which include the players names, connection times, IP addresses, and country of origin. The second file summarises the visits by country, and the last file gets basic info about the server.

Another script then takes these log files and produces an HTML page to be viewed in a browser, which you could then view locally or put up on your web server. Finally, two other scripts produce a shell script which uses GNUplot to produce some charts summarising the data.

To start, grab the tarball from one of the above sources. The tarball will contain the following files:

parselog.py
datatohtml.py
time-chart.py
hour-chart.py
updatelog.sh
logtalk.py
updatetalk.sh
backup-logs.sh
crontab.example

Place these either in the directory where the log files reside, or you can set up a cron schedule to move the log files to a location of you choosing. For example, my Skulltag servers are on a FreeBSD server, and my web server is on an Ubuntu server. I then set up a crontab entry to move the files from one server to the other hourly.

Now, you can use parselog.py to parse the log files and store player-related data into several data files. An example of its use is

$ ./parselog.py doom2.log

After running this, you will get the files

doom2.log.player.dat
doom2.log.country.dat
doom2.log.server.dat

These contain the important parts of the data from the server. Next, you can turn this data into an HTML document for web-viewing by using datatohtml.py:

$ ./datatohtml.py doom2.log.player.dat doom2.log.country.dat doom2.log.server.dat "Doom 2 Coop Server" doom2

The first three arguments are the three data files you just generated. Next, you can give the server a name that will be displayed on the webpage. Finally, repeat the name of the log, but without the .log, for the scripts which produce the graphics.

Finally, if you want charts in your reports, you must have gnuplot installed, as this is what creates the images. For Ubuntu/Debain,

$ sudo apt-get install gnuplot

For FreeBSD,

# cd /usr/ports/math/gnuplot
# make install clean

Now that gnuplot is installed, we can make the charts for the report. The links to the chart have already been placed in the HTML. You can run the charts using

$ ./hour-chart.py doom2.log.player.dat doom2.jpg "Doom2"
$ ./time-chart.py doom2.log.player.dat doom2.time.jpg "Doom2"

The first argument is the player data file, then what you want to call the image. I suggest using .jpg and .time.jpg, as these are what are hard-coded into the python script that create the web page. If you want to change it, simply edit the datatohtml.py script for your liking.

As you can see, this is a lot to type. To automate the task, I have created updatelog.sh, which does all of this. Use it as follows

$ ./updatelog.sh doom2 Doom 2 Coop

The first argument is the name of the log file, without the .log, followed by whatever you want to name the server. I had problems getting string arguments to work, so you can specify the name without quotes after the first argument.

Finally, if you want to let the computer do all of the work, let cron update the logs hourly, daily, or whatever you like. I have included crontab.example as a template,

PATH=/usr/sbin:/usr/local/bin:/usr/bin:/sbin:/bin:/var/www/logs
10 * * * * /var/www/logs/updatelog.sh doom2 Doom 2 Coop > /var/www/logs/fub.out 2>&1
11 * * * * /var/www/logs/updatelog.sh tnt TnT Coop > /dev/null
12 * * * * /var/www/logs/updatelog.sh plutonia Plutonia Coop > /dev/null
13 * * * * /var/www/logs/updatelog.sh doom-ep1 Doom Episode 1 Coop > /dev/null
14 * * * * /var/www/logs/updatelog.sh doom-ep2 Doom Episode 2 Coop > /dev/null
15 * * * * /var/www/logs/updatelog.sh doom-ep3 Doom Episode 3 Coop > /dev/null
16 * * * * /var/www/logs/updatelog.sh nimrod Nimrod Deathmatch > /dev/null
17 * * * * /var/www/logs/updatelog.sh epic2 Epic2 Coop > /dev/null
18 * * * * /var/www/logs/updatelog.sh dwango Dwango Deathmatch > /dev/null
19 * * * * /var/www/logs/updatelog.sh 32in24 32in24 Deathmatch > /dev/null
55 * * * * /var/www/logs/updatetalk.sh > /dev/null


List Posts Newest Posts Page 1Next Page