Scrubbing logs for bad IPs

Periodically, log files should be scrubbed for bad bots and malicious IPs. Let’s do that.

This will work with any log file in which you have one IP address per line.

The first thing we want to do is pull out all the IP addresses in our log file. Let’s go ahead and do that for lawsonry’s log file right now:

grep -E -o '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' >> ipslist.txt

Unfortunately, if you `cat ipslist.txt` you’ll see a bunch of duplicate entries. To get rid of them, let’s sort the file:

`sort -u ipslist.txt >> ipslist`

Now remove our old file with `rm -rf ipslist.txt` because we don’t need it anymore.

Go ahead and `cat ipslist` to see the list of unique IPs.

With this, you can use the methods described at the end of step 5 of this tutorial, in which we’ll create a custom php script to scrub through the IPs and automatically print us out some `deny from` entries formatted for Nginx.

Alternatively, you could go straight to iptables (which is what I do) and just block them at the server level:

iptables -A INPUT -s "$BLOCK_THIS_IP" -j DROP



Written by:

I develop, assess, and implement strategic research solutions for the social sciences by engineering management information systems, designing data mining solutions, and analyzing intelligence through the employment and integration of JavaScript, PHP, MySQL, R, and C/C++ across Nginx, Apache, NodeJS, and many other tools across the stack.

One Comment

    PPR Infotech

    Nice work.

Leave a Reply