Monitor the ACCESS LOGS in order to see who and what bots are hitting you

Posted one year ago

First we need to isolate our access-logs. We can do that with a quick find.

find / -iname 'access-logs'

This should print out a list of access-logs (usually per domain).

Hypothetically “/home/test-domain/access-logs”

Let’s search within out logs now.

grep "<Date/Here>" /file-location | grep "bot"|less -N

The above command looks for a date (22/Sep) within the file for all instances of “bot” we then “less” that and print the line numbers out (-N). We “less” as these files can become quite large and we only want to see a handful of results at a time to give us a manageable chunk of data

Bonus: group IP’s by count

For those of you that are interested in counting the amount of times a particular ip (or phrase) occurs in the access log.

grep 'IP.HERE' /location/of/access-log | cut -d' ' -f1 | sort | uniq -c | sort -r

Sort: sort lines of file
Cut: takes specific columns/chars out of the file given flags
Cut -d: take a whole word as opposed to characters
Cut -d -f1: Combined with -f1 to extract the first column.
Uniq: groups the keyword together, appending “-c” prints out the amount of results… this is the important part!
Finally with a trailing “sort” we list top of the bottom the amount of times the IP appears (-r placing the “most” at the top).