I recently noticed a massive increase in traffic to the website, strangely the traffic was all direct traffic and seem to be coming from one ISP – xlhost.com inc. After a bit of research I found that this ISP is notorious for allowing people to run VPS with bots and scrapers etc and general bad intentions.
I needed to find a way of blocking all traffic from that ISP, which made me think how I can do it, I decided to look at two things – the IP range and also to fight the bots we can block the user agents that where being used to scrape the site. The intention of this post is to provide a guide on how to block these IPs and any other IPs that you may not want to visit your website, over time I will add to this list with your help.
Block User Agents in NGINX
This is very simple to do but only really useful if they offending IP uses there own bot, we need to make sure that access logs are enabled first (click link). Once done view those logs for the offending IP and user agents, I highlight one we will block below.
The part we are interested in is the following:
WeSEE:Ads/PageBot (http://www.wesee.com/bot/
This appears to be a nosey bot trying to see what ads techieshelp runs, its time to make this go away. To do so we need to edit the server block for our website, this is normally located in the following directory and is named after your website.
/etc/nginx/sites-available/
Once you have located your file add the following to your server block, as mention in this example we block WeSEE:Ads/PageBot (http://www.wesee.com/bot.
if ($http_user_agent = "WeSEE:Ads/PageBot (http://www.wesee.com/bot"){
return 403;}
It should look like the below, obviously you can replace the user agent with any offending user agent.
You need to add the above for each user agent, which made me think that blocking IPs or Subnets would be better..
Blocking an IP address or Subnet in NGINX
In this example I block subnets from xlhost.com but over time will add more to the deny file we create. The process below involves creating a deny.conf file and including it in the nginx.conf file in our http block – this keeps everything nice and neat. First lets find the subnets we want to block. In your access log as mentioned above find the offending IP, then use the site to below to get its subnet thanks @linevty !.
http://bgp.he.net/
It will give you the subnet as seen in my example.
Now back on your server run the following command to create deny.conf:
nano /etc/nginx/deny.conf
We are now in this file and we need to add the following to block the subnets we do not want to allow to view our website – here i block most of xlhosts subnets – please send me more if you are aware of them!
deny 209.51.197.0/24; # XLHOST IP deny 209.190.121.32/27; #XLHOST IP deny 209.190.0.0/17; # XLHOST IP deny 173.45.64.0/18; # XLHOST IP deny 64.79.64.0/19; # XLHOST IP deny 64.79.89.0/19; # XLHOST IP deny 64.79.85.0/19; # XLHOST IP deny 207.182.155.96/28; #AXARNET deny 207.182.158.136/29; #AXARNET deny 91.142.208.0/20; #AXARNET deny 95.141.32.0/20; #SeFlow.it deny 82.103.128.0/18; #EASYSPEEDY external network
Once done save this file, we now need to get nginx to call this file. Edit the NGINX.CONF file with the following command.
nano /etc/nginx/nginx.conf
You will presented with a screen similar to below, scroll to the section that says access_log off: and change it to on.
Then locate the HTML block and add the following line in the HTML block, it should look like the example below.
include /etc/nginx/deny.conf;
Save the file and the we need to restart the NGINX service with the following command.
sudo service nginx restart
You have now blocked the subnets from xl host, use the same process to block other offending ips and subnets. If you have had issues with various bots, user agents and nasty IP subnets then please comment below and we can build up a master deny list.
NGINX IP DENY MASTER LIST
deny 209.51.197.0/24; # XLHOST IP deny 209.190.121.32/27; #XLHOST IP deny 209.190.0.0/17; # XLHOST IP deny 173.45.64.0/18; # XLHOST IP deny 64.79.64.0/19; # XLHOST IP deny 64.79.89.0/19; # XLHOST IP deny 64.79.85.0/19; # XLHOST IP deny 207.182.155.96/28; #AXARNET deny 207.182.158.136/29; #AXARNET deny 91.142.208.0/20; #AXARNET deny 95.141.32.0/20; #SeFlow.it deny 82.103.128.0/18; #EASYSPEEDY external network