0

How to disallow one domain path in robots.txt? (there is a module which allows to do that but it is better to avoid it because of performance issues) For example, onedomain.com/path1/* must be disallowed. Is it right to type like this in robots.txt (there is another domain twodomain.com for the same website):

Disallow: onedomain.com/path1/*

Twodomain path must be allowed to be crawled and Onedomain path must be disallowed to be crawled (both of them are pointing to the same path).

Improve this question
6
  • I've read that article. I suppose that my question is a little bit different. They answer about methods which are used to prevent crawling the entier page on both domains but I need that one domain path is crawled and another one mustn't be crawled. I precised that fact in the last paragraph.
    – gladiatorhl2
    Apr 26, 2016 at 18:44
  • Using some form of server rewrite rule, like the answer from StryKaizer on Exclude domain in robots.txt, is probably your best bet.
    – scronide
    Apr 27, 2016 at 0:40
  • I would like to hide not the whole domain but one path.
    – gladiatorhl2
    Apr 27, 2016 at 1:23
  • Once you've split the robots.txt files by domain, you just include Disallow: /path1/ in the file for the first domain.
    – scronide
    Apr 27, 2016 at 17:29
  • Two domains are pointed to the only one database. Is it possible to split the robots.txt in this case?
    – gladiatorhl2
    Apr 28, 2016 at 1:12

1 Answer 1

Reset to default
0

I don't have enough reputation to comment, so I'll just throw this out:

You can test the user agent and the URL in settings.php. Then, if it matches the crawler bot and the /path1/* criteria, you could do a 301 Permanent Redirect for that URL within your settings.php file. Here is how:

  header('HTTP/1.0 301 Moved Permanently');
  header('Location: ' . $base_url . $redirect_path );
  exit();

where $base_url is "Twodomain" and $redirect_path is "/path1/URLfromtest"

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.