2

With the new update of Google algorithm called Penguin, I think my site was being penalized due to webspam. But of course I don't create post which seems to be spam to Google. It is just I think how Google index my site.

I found out that Google index the URL of my site like:

http://www.example.com/comment/reply/3866/26556

So there are so many comment/reply URL index by Google. I have already added:

Disallow: /comment/reply/ Disallow: /?q=comment/reply/

but still Google still index this URL.

Any idea how to prevent Google from indexing comments?

Here's my .htaccess:

<FilesMatch "\.(engine|inc|info|install|make|module|profile|test|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(\..*|Entries.*|Repository|Root|Tag|Template)$">
  Order allow,deny
</FilesMatch>

Options -Indexes
Options +FollowSymLinks
ErrorDocument 404 /index.php
DirectoryIndex index.php index.html index.htm

<IfModule mod_php5.c>
  php_flag magic_quotes_gpc                 off
  php_flag magic_quotes_sybase              off
  php_flag register_globals                 off
  php_flag session.auto_start               off
  php_value mbstring.http_input             pass
  php_value mbstring.http_output            pass
  php_flag mbstring.encoding_translation    off
</IfModule>

<IfModule mod_expires.c>
  ExpiresActive On

  ExpiresDefault A1209600

  <FilesMatch \.php$>
    ExpiresActive Off
  </FilesMatch>
</IfModule>

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule "(^|/)\." - [F]
  RewriteCond %{HTTP_HOST} !^www\. [NC]
  RewriteRule ^ http://www.%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !=/favicon.ico
  RewriteRule ^ index.php [L]

  <IfModule mod_headers.c>
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      Header set Content-Encoding gzip
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>
</IfModule>
  • 2
    You may get a better respone over at webmasters.stackexchange.com. – mpdonadio May 2 '12 at 0:42
3

I would try adding

RewriteCond %{HTTP_USER_AGENT} ^Googlebot
RewriteRule ^comment/reply/$ - [G,L]
RewriteRule ^comment/reply/(.*) - [G,L]

to .htaccess or Apache config to inform spiders that the links should be removed. Just make sure to put it before the catch-all rule at the bottom of the standard Drupal .htaccess. The problem is coming up with a list of spiders to add as [OR] conditions to the RewriteCond so that they get a 410 but Drupal handles normal requests.

Google Webmaster Tools has a URL removal tool, too, but I have never tried using it with patterns before.

  • hi, thank you very much for the information. I don't know where to put it on my .htaccess. please guide me where to put it. I have edited my question to include the .htaccess. – jaypabs May 2 '12 at 1:03
  • also i don't get what you mean by "The problem is coming up with a list of spiders to add as [OR] conditions to the RewriteCond so that they get a 410 but Drupal handles normal requests." – jaypabs May 2 '12 at 1:09
  • 1: try placing the lines after RewriteRule "(^|/)\." - [F]. 2: if Google is indexing the links, it is likely that Yahoo!, Bing, etc, will start indexing them, too. The robots.txt hing should work, but it will take time to heal. – mpdonadio May 2 '12 at 1:18
  • I just got some problem when using this code in .htaccess. the comment does not save and it redirect me to /comment/reply/ after submitting the comment form. please don't use this code. – jaypabs May 3 '12 at 7:03
  • You may need to copy the RewriteCond line to between the RewriteRules so it applies to each of them. – mpdonadio May 3 '12 at 10:45
1

I am facing the same problem at the moment. One possibility is to modify the "reply" link of the main posts, by adding the rel="nofollow" option. This should be doable by writing an external module, or a custom theme function.

The advantage of this method, is that all search engines will avoid following these links, so you do not have to be explicit for google

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.