Scrubbing logs for bad IPs

Periodically, log files should be scrubbed for bad bots and malicious IPs. Let’s do that.

This will work with any log file in which you have one IP address per line.

The first thing we want to do is pull out all the IP addresses in our log file. Let’s go ahead and do that for lawsonry’s log file right now:

grep -E -o '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)' >> ipslist.txt

Unfortunately, if you `cat ipslist.txt` you’ll see a bunch of duplicate entries. To get rid of them, let’s sort the file:

`sort -u ipslist.txt >> ipslist`

Now remove our old file with `rm -rf ipslist.txt` because we don’t need it anymore.

Go ahead and `cat ipslist` to see the list of unique IPs.

With this, you can use the methods described at the end of step 5 of this tutorial, in which we’ll create a custom php script to scrub through the IPs and automatically print us out some `deny from` entries formatted for Nginx.

Alternatively, you could go straight to iptables (which is what I do) and just block them at the server level:

iptables -A INPUT -s "$BLOCK_THIS_IP" -j DROP


About Jesse Lawson

I'm a full stack web developer, graduate student, and writer of really awful science fiction.

1 comment

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>