I've seen yesterday night something which could be described as a "DDOS attack" on one of the infrastructure I'm managing : during approx. 7h we've received a continuous huge amount of connections / HTTP requests coming from more than 1800 differents IP, mainly located in :
- Russian Federation
User-Agent: Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620
After digging into google, it seems that many people had the same experience with this crawler and that requesting rate-limiting was not successful. Moreover, some people also describe the fact that denying 80legs in the robots.txt was not sufficient to prevent them to crawl you. So, in this case I suggest you to put preventive rules either in your Web Application Firewall or in your load-balancer / webserver to prevent them reaching / overloading your web infrastructure.
Below an example of HAProxy configuration to tarpit all HTTP requests from this crawler :
frontend HTTP [...] # # Block all requests from 80legs # reqitarpit ^User-Agent:.*80legs*