7

My website is being DoS'ed by Google webspiders. Google is welcome to index my site, but sometimes it is querying a tagcloud on my site faster than my webserver can produce the results, making my webserver run out of resources.

How can I limit access to my webserver in such a way that normal visitors are not affected?

robots.txt is no option because it would block the whole site from being indexed.

iptables -m recent is tricky, because some pages have a lot of images or other data files and 'recent' triggers on those too (typically my RSS aggregator, loading images and feeds).

iptables -m limit has the same disadvantage and on top of that, I wasn't able to be selective per IP source address.

How can I limit visitors that cause my server load to rise too high?

I am running apache2 on Ubuntu server in a VirtualBox VM.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
jippie
  • 13,756
  • 10
  • 44
  • 64
  • 1
    You can use Webmaster tools to lower the crawl rate. See http://support.google.com/webmasters/bin/answer.py?hl=en&answer=48620 – derobert Oct 08 '12 at 14:34

1 Answers1

5

Try the mod_qos Apache module. The current version has the following control mechanisms.

  • The maximum number of concurrent requests to a location/resource (URL) or virtual host.
  • Limitation of the bandwidth such as the maximum allowed number of requests per second to an URL or the maximum/minimum of downloaded kbytes per second.
  • Limits the number of request events per second (special request conditions). It can also "detect" very important persons (VIP) which may access the web server without or with fewer restrictions.
  • Generic request line and header filter to deny unauthorized operations.
  • Request body data limitation and filtering (requires mod_parp).
  • Limitations on the TCP connection level, e.g., the maximum number of allowed connections from a single IP source address or dynamic keep-alive control.
  • Prefers known IP addresses when server runs out of free TCP connections.

This sample conditional rule from the documentation should get you going in the right direction.

# set the conditional variable to spider if detecting a
# "slurp" or "googlebot" search engine:
BrowserMatch             "slurp"                  QS_Cond=spider
BrowserMatch             "googlebot"              QS_Cond=spider

# limits the number of concurrent requests to two applications
# (/app/b and /app/c) to 300 but does not allow access by a "spider"
# if the number of concurrent requests exceeds the limit of 10:
QS_LocRequestLimitMatch       "^(/app/b/|/app/c/).*$"  300
QS_CondLocRequestLimitMatch   "^(/app/b/|/app/c/).*$"  10   spider
George M
  • 13,589
  • 4
  • 43
  • 53
  • Looks a bit more complicated than I'd hoped for on a friday evening enjoying Belgian beer ... Have to look at it tomorrow. Looks promising with the browsermatch and things. Coming to think of it ... would be nice if it automatically any host / useragent that requests robot.txt :o) Thnx Uther – jippie Apr 27 '12 at 19:58
  • Enjoy that beer! – George M Apr 27 '12 at 20:00
  • OK, it looks pretty simple. Implemented it to see if my webserver wil be able to keep up now. – jippie Apr 29 '12 at 12:57