9

I use Drupal 7, and I am trying to use the "noindex" meta tag to prevent http://example.com/comment/reply/43/738 from being indexed by search engines.

The comment/reply page is being indexed, even though I've used the following role in robots.txt:

Disallow: /comment/reply/

In which template file will I need to incorporate "noindex" meta tag for /comment/reply/ pages?

  • 1
    What rule have you written in your robots.txt file ? – tostinni Dec 26 '11 at 18:59
  • @tostini:I've used Disallow: /comment/reply/ – Naweed Chougle Dec 27 '11 at 13:48
7
+50

You can use any of this code in robots.txt:

User-agent: *
Disallow: /*comment

This will neglect every URL that contains comment.

You can also use the following, to neglect every URL containing /comment.

User-agent: *
Disallow: /comment/reply
User-agent: *
Disallow: /comment

After you do this, check whether it is working or not by using the Google Robots.txt checker.

|improve this answer|||||
  • What do you think about this? google.co.in/support/forum/p/Webmasters/… – Naweed Chougle Dec 27 '11 at 13:57
  • Since Noindex is the meta tag it will prevent the whole page from being indexed.Now you have disabled the comments from crawled now so you can remove the indexed links from Google using webmaster tools. go to site configuration >> crawler access >> Remove URL .I think this will be the better solution. – Vivek R Dec 29 '11 at 17:28
  • After I remove the URLs using webmaster tools, new ones with the same pattern /comment/reply/ appear! And the URLs were indexed even though the Disallow: /comment/reply/ was being used in the robots file. – Naweed Chougle Dec 29 '11 at 17:43
  • 2
    give some time for the search engine to remove..it is not going to happen overnight so wait for some time..may be a week or so. – Vivek R Dec 29 '11 at 17:58
  • As I said earlier, the URLs were being indexed even after I used Disallow /comment/reply/ ....This could be happening because of links to the reply form coming from external sites. I've updated my question and included details about the possible need for the meta noindex tag. – Naweed Chougle Dec 30 '11 at 13:44
4

In your theme's template.php, or in page.tpl.php, you can check the page's url to see if the page is a comment page, then add code that will insert the meta tag.

You can do this in the template.php function YOURTHEME_preprocess_html(), or insert it somewhere near the top of page.tpl.php. The code to add would be something like this:

    <?php
      $element = array(
        '#tag' => 'meta', 
        '#attributes' => array(
          'property' => 'robots',
          'content' => 'noindex',
        ),
      );
    drupal_add_html_head($element, 'robots');
?>

See drupal_add_html_head().

|improve this answer|||||
  • Thanks, @jmarkel! Can you help me check, programmatically, that the page is something like: example.com/comment/reply/32/409 ....so that I can set the meta noindex only for such pages? – Naweed Chougle Jan 2 '12 at 7:47
  • 1
    Here's the problem - there really isn't any such node type as a comment - they only seem to live with the nodes they are commenting on. So it's not really possible to generically distinguish them. If you click a comment's permalink, you get a url that starts with 'comment/' but that doesn't help because they don't carry that url internally. I suppose you could look for the "page" argument on subsequent pages (i.e. when there's more than one page of comments) but you're still left having to distinguish between comments and other pages using a pager. Long story short - not sure how to do it... – jmarkel Jan 3 '12 at 2:20
  • For now, @jmarkel, I'll have to live with Google webmaster's URL removal tool. Going by one of Matt Cutts videos, it seems that a URL once removed will not reappear. The problem is that I've seen new ones appearing in the search results! – Naweed Chougle Jan 3 '12 at 4:42
1

Update: the following works, it prints <meta name="robots" content="noindex,follow" /> in the head section of all pages accessed via a path starting with comment. This is to work around the fact that a page like comment/3 internally carries node/nid as arguments, as jmarkel pointed out above.

<?php
function metarobots_comment_help() {
  $url_components = explode('/', request_uri());
  if ($url_components[1] == 'comment') {
    $elements = array(
      '#tag' => 'meta',
      '#attributes' => array(
        'name' => 'robots',
        'content' => 'noindex,follow',
      ),
    );
   drupal_add_html_head($elements, 'robots');
  }
}

With the aid of a helpful comment on the arg function.

Hopefully this will eventually be addressed by Meta Tags module - there's a feature request, but it's not yet clear if the module author wants to support this.


[My previous comment] I just wanted to add that using robots.txt is not the answer - as you already noticed, links keep appearing in the search results despite using Disallow: /comment. This is expected, since robots.txt tells the bot not to crawl those pages, but it doesn't tell Google not to index it. As explained on SEOmoz Robots.txt and Meta Robots Best Practices for Search Engine Optimization:

In most cases, meta robots with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation.

Block with Robots.txt - This tells the engines to not crawl the given URL but tells them that they may keep the page in the index and display it in in results.

Block with Meta NoIndex - This tells engines they can visit but they are not allowed to display the URL in results. (This is the recommended method) So the noindex meta tag you're after is indeed what you need.

|improve this answer|||||
  • That's true, @arjan, you've restated one of the comments I made with extra details. I'm looking for a way to set meta noindex for the pages I don't want in the search results. – Naweed Chougle Jan 4 '12 at 1:50
  • 1
    Yes, it seemed you weren't 100% certain so I wanted to add that it's correct ;) – arjan Jan 4 '12 at 14:48
  • @ProgrammingEnthusiast: see update above. – arjan Jan 14 '12 at 19:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.