3

I'm being trolled by China, and don't know why I can't block their request to my server.

//host.deny

ALL: item.taobao.com
ALL: 117.25.128.*

But when I watch the error log on my webserver tail -f /var/log/apache2/error.log the requests are still being allowed through.

Question: Why isn't my /etc/hosts.deny config working?

DanSut
  • 592
  • 4
  • 14
Jordan Davis
  • 141
  • 1
  • 6

2 Answers2

11

  1. The file is called /etc/hosts.deny, not host.deny

  2. Not all services use tcp-wrappers. sshd, for example, doesn't by default. Neither does apache.

  3. You can use iptables to block all packets from 117.25.128/24, e.g.:

     iptables -I INPUT -s 117.25.128.0/24 -j DROP

  4. Even better, you can use fail2ban to monitor a log file (such as apache's access.log and/or error.log) and automatically block IP addresses trying to attack your server.

From the debian fail2ban package description:

Fail2ban monitors log files (e.g. /var/log/auth.log, /var/log/apache/access.log) and temporarily or persistently bans failure-prone addresses by updating existing firewall rules.

Fail2ban allows easy specification of different actions to be taken such as to ban an IP using iptables or hosts.deny rules, or simply to send a notification email.

By default, it comes with filter expressions for various services (sshd, apache, qmail, proftpd, sasl etc.) but configuration can be easily extended for monitoring any other text file. All filters and actions are given in the config files, thus fail2ban can be adopted to be used with a variety of files and firewalls.

muru
  • 69,900
  • 13
  • 192
  • 292
cas
  • 1
  • 7
  • 119
  • 185
  • Ok yea I'll edit the tile. Ah okay, well I tried `iptables -A INPUT -s 117.25.128.0/24 -j DROP` but that didn't block it either. – Jordan Davis Nov 17 '15 at 04:10
  • 1
    `-I INPUT` inserts the new rule at the top of the INPUT chain. `-A INPUT` adds it to the bottom of the chain. if `-A INPUT` didn't work, there must have been some other rule earlier in the chain which allowed those packets. – cas Nov 17 '15 at 04:11
  • ahhh okay, I'm assuming precedence is top to bottom then? – Jordan Davis Nov 17 '15 at 04:12
  • yep. first matching rule wins. – cas Nov 17 '15 at 04:13
  • Perfect it looks to be working! Thank you. Quick question though so alot of the requests are coming from the same refer ie. (item.taobao.com) with different client IP's, so I'm assuming they're using some type of redirect. Is there a way to block by referer, or block by 117.* using like a wildcard selector? – Jordan Davis Nov 17 '15 at 04:30
  • not easily. more to the point, not cheaply if you're trying to conserve bandwidth and/or CPU power. there are iptables that can inspect the contents of a packet to make decisions on. and apache can be configured to reject requests with matching Referer: headers. You can even have apache redirect to a special page for requests with unwanted Referers (you'd need to enable mod_rewrite to do this) – cas Nov 17 '15 at 04:43
  • Makes sense, yea I'll just configure a directive on Apache for this application. But, for an Enterprise application your saying I would probably want to use a few proxy servers to open/read the packets and filter the requests before forwarding them to my application server, so it's on a separate server and doesn't suck my application server CPU/bandwidth? – Jordan Davis Nov 17 '15 at 04:55
  • Well, you could do that but i didn't say it. I'd just use fail2ban to drop packets from annoying client IPs...I don't see any reason to accept and process packets from source IPs that have proven themselves to be a nuisance. – cas Nov 17 '15 at 05:04
  • Perfect thanks man yea I'll take a look at fail2ban looks good. – Jordan Davis Nov 17 '15 at 05:55
4

As for your original question. My Apache in debian is not configured with libwrap, so it won't consult hosts.deny. [the previous answer already mentions it - the reality is that tcpwrappers is not the epitome of security as it was in the 90s, specially when it comes to blacklisting]. You would have to run it not as a daemon, but from (x)inetd, which would slow it down (considerably).

You can block/allow access at Apache level, and do not need tcp wrappers for Apache [nor iptables for that matter]. You have not mentioned ssh, however I never leave ssh servers open to the outside directly. However keep reading on.

We have a vhost with 300+ domains, and also similar problems, with taobao, baidu, and often even with google spiders. Notably baidu spiders can be quite aggressive and persistent.

As you have already figured out, they have farms of servers, and even if you block an IP they will appear again shortly from some other IP addresses.

It is not practical at all to try and maintain lists of IP addresses/netblocks by hand.

What it works for us rather well is modsecurity blocking user agent strings permanently while mod_evasive is blocking temporarily IPs that are being abusive.

This setup, besides slowing down spiders from search engines, also has the advantage of throttling down zombies trying to guess passwords on CMSes.

The relevant part of our modsecurity.conf

SecRule REQUEST_HEADERS:User-Agent "Yandex" phase:1,deny,nolog,id:'6972'
SecRule REQUEST_HEADERS:User-Agent "ichiro" phase:1,deny,nolog,id:'6973'
SecRule REQUEST_HEADERS:User-Agent "Baiduspider" phase:1,deny,nolog,id:'6974'
SecRule REQUEST_HEADERS:User-Agent "Baiduspider/.*" phase:1,deny,nolog,id:'6975'
SecRule REQUEST_HEADERS:User-Agent "Baiduspider-video" phase:1,deny,nolog,id:'6976'
SecRule REQUEST_HEADERS:User-Agent "Baiduspider-image" phase:1,deny,nolog,id:'6977'
SecRule REQUEST_HEADERS:User-Agent "sogou spider" phase:1,deny,nolog,id:'6978'
SecRule REQUEST_HEADERS:User-Agent "YoudaoBot" phase:1,deny,nolog,id:'6979'
SecRule REQUEST_HEADERS:User-Agent "bingbot(at)microsoft.com" phase:1,deny,nolog,id:'6980'
SecRule REQUEST_HEADERS:User-Agent "msnbot(at)microsoft.com" phase:1,deny,nolog,id:'6981'
SecRule REQUEST_HEADERS:User-Agent "BLEXBot/1.0" phase:1,deny,nolog,id:'6982'
SecRule REQUEST_HEADERS:User-Agent "Bot.*" phase:1,deny,nolog,id:'6984'
SecRule REQUEST_HEADERS:User-Agent "AhrefsBot.*" phase:1,deny,nolog,id:'6985'

And our mod-evasive.conf

DOSHashTableSize 2048
DOSPageCount 10
DOSSiteCount 300
DOSPageInterval 2.0
DOSSiteInterval 1.0   
DOSBlockingPeriod 600.0   
DOSLogDir /var/log/apache2/evasive
DOSWhitelist 127.0.0.1
DOSWhitelist 1xx.xxx.xxx.xx

I also forgot a very real possibility. If you do not deal with China or are running your home server, just block the whole country. The level of attacks and malware that comes from them has justified many professionals to do that.

http://www.cyberciti.biz/faq/block-entier-country-using-iptables/

I also forgot to add to this rather lengthy answer a footnote. Often people suggest me at work to use robots.txt for these kind of problems. The point is robots.txt is only a suggestion for remote programs.Rogue actors certainly ignore them, and it is not guaranteed other web crawlers honor them nowadays. From our tests, for instance, Baidu seems to not honor them. (robots.txt it tantamount to ask a gangster, please tickle me instead of punching me)

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
  • 1
    Thanks man! yea I'm def going to use mod_evasive, those configure file settings work good for you? – Jordan Davis Nov 17 '15 at 05:54
  • They work quite well, most of our vhosts do not have much visitors. Otherwise you have to increase dospagecount (max visits to a single page from an IP), and dossitecount (max visits of and IP) accordingly. – Rui F Ribeiro Nov 17 '15 at 06:08
  • Look to my first lines again please. – Rui F Ribeiro Nov 17 '15 at 06:11
  • Got it makes sense, I'll read up on the settings. Thanks again for all your help! – Jordan Davis Nov 17 '15 at 06:14
  • (do not take it wrongly, just suggesting blocking addresses in web server or Apache could be a better title)...nevertheless, it was not for this comment that I came here again, read the answer again please. Besides comments in the beggining, you will find something more interesting in the end. – Rui F Ribeiro Nov 17 '15 at 06:40
  • Came here to add a few comments about hosts.txt as just today I had a conversation about them with a co-worker, about a supplier using them and about their ineffectiveness. They are certainly relevant to complete the post. – Rui F Ribeiro Nov 18 '15 at 13:40
  • thanks for the detailed explanation. I've just spent the last day or so writing a script which blocks China from my web server and Ive read this post with alot of interest. I also just asked this question about [whether or not rate limiting on firewalls is effective](http://security.stackexchange.com/questions/105906/how-effective-are-firewall-rate-limiting-rules-for-slowing-down-brute-force-and). After reading this post Ive come to the conclusion that mod security is better than firewall for rate limiting HTTP requests - because you can define rules per web pages/URL and not TCP connections. – the_velour_fog Nov 19 '15 at 13:20
  • rate limiting and firewalls have their uses. For instance, having firewalls at the front of my servers, allow me to alleviate the security configurations. The configuration of my home server for instance, is radically different from my virtual farm of servers at work in what touches to security. Ultimately everything is built in building blocks, and often security has to balance usability. I could swear there are packages to block countries, I know FreeBSD has them, and I *think* Linux has at *least* ready made scripts. – Rui F Ribeiro Nov 19 '15 at 13:56
  • Yes the script I just wrote adds IP addresses to the firewall, its not alot of code, but it little tricky organising so the rules persist on reboot and then scheduling automatic updates of IP blocks. Im currently trying to work out if it possible to apply your mod evasive and mod security config to nginx – the_velour_fog Nov 19 '15 at 14:32
  • research limit_conn_zone for nginx – Rui F Ribeiro Nov 19 '15 at 19:01
  • thanks! thats very cool it's based on number of requests as well as connections and implemented natively by nginx, thanks for the info! – the_velour_fog Nov 20 '15 at 00:24