5

Is it possible to exclude a domain name in robots.txt?

I have a single Drupal site with multiple domains pointing on it. For example our dev and staging server are being crawled by Google when our .htaccess password protection is disabled.

I don't intent to block any access in our development server but to tell search engine bots to stop indexing all the contents when the domain is/are dev or staging.

Basically I am seeing some of the links pointing in dev servers when I search some of my content in Google.

Is there any way or any module that could handle this?

EDIT:

Here's how I ended. Inside theme_preprocess_html().

// Avoid search engines to index dev and stage domains.
  $restricted_domains = array('dev.mydomain.com', 'stage.mydomain.com', 'editorial.mydomain.com'); 
  if (in_array($_SERVER['HTTP_HOST'], $restricted_domains)) {
    $meta_robot = array(
      '#tag' => 'meta',
      '#attributes' => array(
        'name' => 'robots',
        'content' => 'noindex, nofollow'
      ),
    );
    drupal_add_html_head($meta_robot);
  }
  • 1
    This is a great solution. We have too many environments and too many external API consumers to set up a "Proper Firewall" on every one of them (staging and qa servers), and really we arent trying to prevent external access to our servers, we're trying to prevent content from being indexed into major search engines, which are all well behaved. – Mixologic Jun 4 '13 at 0:07
9

Robots.txt will not help your site being crawled (technically) by anything. They're just guidelines which Bots should follow. You should limit dev and test server access by Firewall rules, or as you've mentioned by httpauth.

| improve this answer | |
  • How abut using this technique <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">? Is there any fallback in this method? Basically I will add that tag when the base URL is not equal to public domain. – ninjascorner Apr 26 '13 at 2:41
  • 2
    essentially for Google yes this will likely stop google from indexing your content. but your mixing apples with oranges; if you dont want remote access to your site, there are better ways to do it. Put another way your dev/staging/test servers should be mirrors of production and usually not have magic bits that make them treat content differently. – tenken Apr 26 '13 at 4:01
2

Add these lines to your .htaccess:

# Dev environments other robots.txt to prevent indexing.
RewriteCond %{HTTP_HOST} (.*)mystagingurl\.com
RewriteCond %{REQUEST_URI} (robots\.txt)
RewriteRule .* robots-dev.txt [L]

Create an extra file in your root, named robots-dev.txt, and add following code

User-agent: *
Disallow: /

This will serve another robots.txt on the matching domains, excluding them from all search engines which follow the robots.txt-guidelines.

Some examples of matching urls for the given example are:

  • mystagingurl.com
  • www.mystagingurl.com
  • abc.mystagingurl.com
  • abc.def.mystagingurl.com
| improve this answer | |
1

As stated by tenken, robots.txt or robots metatags does not block robots. They are just guidelines. However, all well-behaved robots (and googlebot falls in this category) will obey these guidelines.

Assuming (based on what you say in your question) that you have a production domain named mydomain.com, and three subdomains you do not want appearing in Google's SERP named dev.mydomain.com, stage.mydomain.com and editorial.mydomain.com, having the followming robots.txt in the root directory (DocumentRoot) of the three subdomains you want to exclude will work:

User-agent: *
Disallow: /

This is assuming that the production domain (mydomain.com) do not have the same DocumentRoot as the development sub-domains (dev.mydomain.com, stage.mydomain.com and editorial.mydomain.com). If your web-sever is configured so that all four domains are just aliases and point to the same root directory, then the roobots.txt file will be shared among all four, and will of course also prevent the main site from being crawled. In that case, you cannot use robots.txt, but must resort to adding metatags to the header depending upon domain, as you describe in the most recent version of your question.

Comment: It is correct, as staten by tenken, that using robots.txt or metatags is not the best way to protect development and staging servers from public access - but using this should work as a quick fix until you've put up a proper firewall.

| improve this answer | |
  • I only have single htaccess and this is a single Drupal installation. User-agent: * Disallow: / will also affect the production. I dont intend to block the robots but I want them to not index the site. We are implementing httpauth in our dev and staging but sometimes we are turning it off for testing purposes. See my approach in my original post and feel free to correct me if I'm wrong. – ninjascorner Apr 26 '13 at 6:05
0

I solved it with the metatags module (https://www.drupal.org/project/metatag). In dev.mydomain.com I configured the global settings of the metatags module (/admin/config/search/metatags/config/global) accordingly.

In the global settings click on "Advanced" and check "Prevent search engines from indexing this page." under "Robots".

This will add

<meta content="noindex" name="robots">

to the section of your development site.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.