Too many hits on webserver

My webserver often gets a high CPU load. (often = maybe once a week). Poking around I’ve discovered its random internet actors just poking around via scripts either for weaknesses, or I guess for indexing. Personally I think its vulnerability probing, the big web indexers tend to be much softer in their scanning. The easy and surprisingly very effective thing is to grab the IP, do a whois to get the netblock, and then just iptable drop the whole subnet. Assuming its a non regular client that should be looking at your stuff. If its a reagional ISP that might actually have a potential student or faculty, you have to be a little more pinpoint and block the IP only. If I block all of RoadRunner, I might get a lot of calls from angry students!

Handy shell script to see most hits on your web server from the access logs (lots of them, this server has 68 sites on it)

cat *access_log | cut -f 1 -d \  | sort -n | uniq -c | sort -n

When I ran this earlier today, almost all the hits were in the single digits up to 100’s. Then two outliers in the 50k range. Those two were some random ISP in Germany. I don’t think we have any exchange students from Germany so I just blocked the entire /20. If anyone complains, I’ll remove it. Load went from 12 back to the usual 0.5.