|
I appear to be getting quite heavily crawled by robots that are seeing urls like ?q=node/668 1) how are robot's able to find these urls? The robots are referring too are not google or popular search engines. Are there anyway apart from robots.txt to defend against these. The amount of traffic i'm getting is quite intense. I'm using drupal 7 |
|||||
|
|
Robots can find urls by following links either on-site or offsite. You could use a link scanner as suggested by Clive, or look in google analytics to see where referrals are coming from to see if you have a lot of inbound links. Alternatively, if a robot has figured out (or guessed) that your site is Drupal based then it's a pretty simple matter to start trying ?q=node/1, ?q=node/2, etc... and see what you don't get a 404 from. I'd say a likely reason that the robot is using "?q=blah" instead of just "blah" is because the former will always work whereas the latter requires you to have clean URLs configured. As well as robots.txt there are free services like Cloud Flare http://www.cloudflare.com/ that attempt to filter known-malicious bots for you before they can access your server. You could also add rel="nofollow" to links on you're site that you're suspicious might be generating the traffic, which tells crawlers to not follow that link. Be careful with this as it also stops search engines, meaning it can harm your SEO if used unwisely. |
|||
|
|
