0

Is it possible to block internal Drupal paths for anonymous users and leave only paths generated by pathauto? For example:

  • block path: /taxonomy/term/5
  • leave path: /products/books

In google webmaster tools for my site, all taxonomy pages have an issue called "Pages with duplicate title tags" and google is showing both paths for each term, which is a problem. How to block the internal Drupal path for anonymous users?

By the way, apart from this, some taxonomy terms are purely for config and their paths should be blocked not because googlebot has a problem but generally, those paths make no sense for non-admins and should never be accessed directly via url.

Improve this question
3
  • You shouldn't need to block those paths, Drupal includes a meta canonical by default. If those are missing from your pages' <head> section, it'd be a better idea to find out why, and reinstate them – Clive♦ Oct 30 '13 at 18:14
  • Are you using Global Redirect? – mpdonadio♦ Oct 30 '13 at 18:16
  • Yes, I'm using GD and it's redirecting ok to the pathauto paths. There is also <link rel=canonical...> tag on each taxonomy page with correct path. But still, there are those "duplicate title" html issues for all taxonomy terms - maybe Gbot remembers those paths from their initial, pre-pathauto state (but I installed pathauto a few weeks ago and the "duplicate title" date is Oct 28, so..). Maybe as long as the internal paths are crawlabe, they will generate the issues. I've blocked them in robots.txt as suggested in the answer and check whether the problem is gone tomorrow (it should). – camcam Oct 30 '13 at 22:32
0

To prevent those pages from being crawled, add the following to your robots.txt:

 Disallow: /taxonomy/
 Disallow: /taxonomy/*

For more details, see https://drupal.org/node/22265

Alternatively, you can also use the module disable_term_node_listings to hide node listings for a paricular taxonomy vocabulary.

Improve this answer
2
  • 1
    Historically, some spiders have not treated wildcards in paths very well. If you specify /taxonomy/ then that really tells the spider to not go into that directory. – mpdonadio♦ Oct 30 '13 at 18:21
  • Yeah, it depends on the particular crawler - stackoverflow.com/a/4833663 for details. Google specifically respects it, though. Re-ordered to reflect preference. – jonpeck Oct 30 '13 at 18:25
1

If you have clean URLs enabled, and are using the Global Redirect module, the users should never actually see the default paths (sometimes also called the internal path).

If a user goes to node/123 or taxonomy/term/456, and a path alias exist, then they will get an HTTP 301 and then redirect the the path alias. Google and won't index the paths that 301.

In addition, Drupal will add a rel='canonical' to the header, to notify the world about the offical URL.

However, if you ever had you site without Global Redirect enabled, then you could at one point access pages from multiple paths. Once Google starts seeing 301 responses for paths, it will eventually delist the page. You can speed up the process by also disallowing the paths in your robots.txt and/or submitting page/directory removal requests. Just remember to backup your robot.txt (or better yet put it in your VCS) because it will get nuked during core updates.

Improve this answer
0

When I did this, the only way I could find to make this work without breaking core Drupal functionality was to use LocationMatch inside the Apache vhost configuration

<LocationMatch ^/taxonomy/term>
   SetHandler server-status
   Order Deny,Allow
   Deny from all
</LocationMatch>
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.