Take the 2-minute tour ×
Drupal Answers is a question and answer site for Drupal developers and administrators. It's 100% free, no registration required.

I appear to be getting quite heavily crawled by robots that are seeing urls like

?q=node/668

1) how are robot's able to find these urls?

The robots are referring too are not google or popular search engines. Are there anyway apart from robots.txt to defend against these. The amount of traffic i'm getting is quite intense.

I'm using drupal 7

share|improve this question
1  
It's almost impossible to say without being able to inspect the site...run a link scanner on your site and see which pages refer to those paths, that's the easiest solution. If there aren't any, then essentially you've got bots that are 'guessing' based on some criteria or other, and there's not much you can do about it short of banning their IPs –  Clive Jan 17 '13 at 13:00
add comment

1 Answer

up vote 1 down vote accepted

Robots can find urls by following links either on-site or offsite.

You could use a link scanner as suggested by Clive, or look in google analytics to see where referrals are coming from to see if you have a lot of inbound links.

Alternatively, if a robot has figured out (or guessed) that your site is Drupal based then it's a pretty simple matter to start trying ?q=node/1, ?q=node/2, etc... and see what you don't get a 404 from.

I'd say a likely reason that the robot is using "?q=blah" instead of just "blah" is because the former will always work whereas the latter requires you to have clean URLs configured.

As well as robots.txt there are free services like Cloud Flare http://www.cloudflare.com/ that attempt to filter known-malicious bots for you before they can access your server.

You could also add rel="nofollow" to links on you're site that you're suspicious might be generating the traffic, which tells crawlers to not follow that link. Be careful with this as it also stops search engines, meaning it can harm your SEO if used unwisely.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.