7

Say we have a node called node/123 with comments enabled. When comments are made, additional URLs are generated /comment/1, /comment/2 ... and /comment/reply/2/1 etc. Unfortunately, these new URLs are a page that contains a full copy of the content in the original page /node/123 or the comment in reply. Core's robots.txt only has Disallow: /comment/reply/

Would there be any implications to adding the following to the robots.txt file? Would this be recommended (or even post a core issue pointing this problem out?)

# Paths (clean URLs)
Disallow: /comment/
Disallow: /comment*/
Disallow: /comment/reply/
Disallow: /comment/reply*/
...
# Paths (no clean URLs)
Disallow: /?q=comment/
Disallow: /?q=comment*/
Disallow: /?q=comment/reply/
Disallow: /?q=comment/reply*/
Improve this question
3
  • 2
    There is a similar discussion here: drupal.org/node/1680978 also check out this sandbox module : Permalinks to Nodepath. drupal.org/sandbox/Ayesh/1578662
    – gilzero
    Commented Oct 27, 2012 at 3:00
  • 1
    I would actually take this somewhere else, as this is a general SEO problem, and only loosely related to Drupal.
    – Letharion
    Commented Nov 26, 2012 at 18:24
  • 2
    I initially considered taking this somewhere else, but this is a problem that is specific to Drupal. It probably needs someone to chime in that is familiar with SEO and knows Drupal.
    – Nigel Waters
    Commented Nov 27, 2012 at 4:08

2 Answers 2

Reset to default
5
+500

I'm the author of that sandbox module(Permalink to Nodepath) mentioned in the first comment for the question. Mario's answer is well described regarding canonical URLs - but unfortunately it's not the only implication.

These comment/* URLs are from Drupal 7. In Drupal 6, comments did not have their own URLs. Canonical URLs can solve the problem with duplicate contents which is perfectly fine for most sites IMO. But spam comments can add 404 and 403 errors to your site errors list as they wish. This was my personal experience that my spam hunting service couldn't delete some of the comments so I had to manually delete them. But Google had them already indexed.

screenshot of my blog's crawl errors from Google Webmaster Central

The actual problem is that anonymous users can post any type of comments and if they were deleted or unpublished, Google (and other search engines) logs them as crawl errors which is quite annoying and probably it will affect your page rank as well.

In other hand, most link sharing sites, including facebook do not respect the canonical URL. For an example, http://example.com/node/1 can have 56 (facebook) likes but they are not synced if the user liked http://example.com/comment/5 (assume comment 5 is a comment made to node 1). From facebook view, they are 2 different URLs (actually they are) so like counts are different.

Also, modules like Boost rely on the actual request URI setting from $_SERVER which is still comment/5 (even though canonical URL is node/5). You can however change configuration of Boost module to overcome this.

It's quite popular to use arg(1) to get the node ID in custom blocks, modules, Views, etc. They will work, because in the callback function of comment/% URLs, that function sets $_GET['q'] to the canonical URL. This $_GET change solves most of the problems in progrmable stuff.

Improve this answer
5

After investigating, I don't think there's an SEO problem here. It's an annoyance for sure but it's not a problem and here's why:

As per Google's recommendations here http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html on canonical URLs and duplicate content, you will not have any problems if you have duplicate content only if you properly specify your canonical URLs. Also, Google (and most probably all other search engines) will only display the canonical URL in search results and not the other URLs.

In Drupal's case, the canonical URLs are properly set. For example, I have created a test article (node/2) and added comments and replies to it and got the following pages:

  • /node/2
  • /comment/2
  • /comment/3

After inspecting their generated source code, they all have the same <link rel="canonical" href="/node/2" /> in their source code properly created. So basically only the main article page will display in the search results.

Now, whether to modify core to only generate a single page or to have the robots.txt file patched as you specified is beyond me. I'm sure there will be lots of considerations and special cases to consider before implementing any of the above and since there really isn't a problem maybe it's best to keep things as is.

Cheers!

Note: I'm using Drupal 7.17 (latest version as of today).

Improve this answer
4
  • I think this is definitely the correct answer. node_page_view() sets the canonical URL (api.drupal.org/api/drupal/modules%21node%21node.module/function/…) so there's not much else to worry about. If you don't like the /comment/% callbacks you could definitely add them to robots.txt or eliminate them altogether via hook_menu_alter().
    – Charlie Schliesser
    Commented Nov 27, 2012 at 16:40
  • 2
    I would also like to add that you can already hit a URL like example.com/node/123, /node/123/view/foo, /node/123/view/foo/bar/baz, and all return 200 – setting the canonical URL helps in all of these kinds of instances.
    – Charlie Schliesser
    Commented Nov 27, 2012 at 16:42
  • I'm having the same problem, but my problem is canonical url's are enabled but NOT pointing to my node page - instead, my canonicals are pointing to the comment/replay page. Where to go to edit so they are pointing correctly?
    – blue928
    Commented Jul 15, 2013 at 10:38
  • Weird! I don't think there's an admin page where you can go and edit those. Which Drupal version are you using and what modules do you have installed?
    – Mario Awad
    Commented Jul 17, 2013 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.