Google keeps indexing /comment/reply URL

Question

With the new update of Google algorithm called Penguin, I think my site was being penalized due to webspam. But of course I don't create post which seems to be spam to Google. It is just I think how Google index my site.

I found out that Google index the URL of my site like:

http://www.example.com/comment/reply/3866/26556

So there are so many comment/reply URL index by Google. I have already added:

Disallow: /comment/reply/ Disallow: /?q=comment/reply/

but still Google still index this URL.

Any idea how to prevent Google from indexing comments?

Here's my .htaccess:

<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(\..*|Entries.*|Repository|Root|Tag|Template)$">
  Order allow,deny
</FilesMatch>

Options -Indexes
Options +FollowSymLinks
ErrorDocument 404 /index.php
DirectoryIndex index.php index.html index.htm

<IfModule mod_php5.c>
  php_flag magic_quotes_gpc                 off
  php_flag magic_quotes_sybase              off
  php_flag register_globals                 off
  php_flag session.auto_start               off
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_flag mbstring.encoding_translation    off
</IfModule>

<IfModule mod_expires.c>
  ExpiresActive On

  ExpiresDefault A1209600

  <FilesMatch \.php$>
    ExpiresActive Off
  </FilesMatch>
</IfModule>

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule "(^|/)\." - [F]
  RewriteCond %{HTTP_HOST} !^www\. [NC]
  RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !=/favicon.ico
  RewriteRule ^ index.php [L]

  <IfModule mod_headers.c>
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      Header set Content-Encoding gzip
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>
</IfModule>

Answer 1

I would try adding

RewriteCond %{HTTP_USER_AGENT} ^Googlebot
RewriteRule ^comment/reply/$ - [G,L]
RewriteRule ^comment/reply/(.*) - [G,L]

to .htaccess or Apache config to inform spiders that the links should be removed. Just make sure to put it before the catch-all rule at the bottom of the standard Drupal .htaccess. The problem is coming up with a list of spiders to add as [OR] conditions to the RewriteCond so that they get a 410 but Drupal handles normal requests.

Google Webmaster Tools has a URL removal tool, too, but I have never tried using it with patterns before.

Answer 2

I am facing the same problem at the moment. One possibility is to modify the "reply" link of the main posts, by adding the rel="nofollow" option. This should be doable by writing an external module, or a custom theme function.

The advantage of this method, is that all search engines will avoid following these links, so you do not have to be explicit for google