0

This question already has an answer here:

How to disallow one domain path in robots.txt? (there is a module which allows to do that but it is better to avoid it because of performance issues) For example, onedomain.com/path1/* must be disallowed. Is it right to type like this in robots.txt (there is another domain twodomain.com for the same website):

Disallow: onedomain.com/path1/*

Twodomain path must be allowed to be crawled and Onedomain path must be disallowed to be crawled (both of them are pointing to the same path).

marked as duplicate by Shawn Conn, Adrian Cid Almaguer, No Sssweat, Krishna Mohan, googletorp May 5 '16 at 8:43

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • I've read that article. I suppose that my question is a little bit different. They answer about methods which are used to prevent crawling the entier page on both domains but I need that one domain path is crawled and another one mustn't be crawled. I precised that fact in the last paragraph. – gladiatorhl2 Apr 26 '16 at 18:44
  • Using some form of server rewrite rule, like the answer from StryKaizer on Exclude domain in robots.txt, is probably your best bet. – scronide Apr 27 '16 at 0:40
  • I would like to hide not the whole domain but one path. – gladiatorhl2 Apr 27 '16 at 1:23
  • Once you've split the robots.txt files by domain, you just include Disallow: /path1/ in the file for the first domain. – scronide Apr 27 '16 at 17:29
  • Two domains are pointed to the only one database. Is it possible to split the robots.txt in this case? – gladiatorhl2 Apr 28 '16 at 1:12
0

I don't have enough reputation to comment, so I'll just throw this out:

You can test the user agent and the URL in settings.php. Then, if it matches the crawler bot and the /path1/* criteria, you could do a 301 Permanent Redirect for that URL within your settings.php file. Here is how:

  header('HTTP/1.0 301 Moved Permanently');
  header('Location: ' . $base_url . $redirect_path );
  exit();

where $base_url is "Twodomain" and $redirect_path is "/path1/URLfromtest"

Not the answer you're looking for? Browse other questions tagged or ask your own question.