3

The default robots.txt shipped with Drupal contains rules such as this:

Disallow: /user/register/

If I understand correctly, the leading slash means this really has to match the beginning of the path. E.g. a path like /xyz/user/register/ would not match the rule. (Please correct me if I'm wrong)

(For the ending slash I am not sure)

With the language path prefix option, Drupal will have urls like /fr/user/register etc.

It can be assumed that if we want to exclude /user/register, we also want to exclude /fr/user/register and other language versions.

However, a /*/user/register/ might be too generic, as it also would match other urls which we don't want to exclude.

Questions:

  • Will Drupal's default robots.txt also exclude language-specific versions of the targeted urls? (according to my own assumption it does not).
  • What would be the simplest / cleanest way to exclude all language versions for a given path?

I think this applies equally to Drupal 7 and Drupal 8.

Improve this question
2
  • 3
    Hmm, never thought of this before...looks like I've got some sites to fix ;) You're right that crawlers won't (or at least are very unlikely to) decipher the language prefix from the rest of your site and apply it to the rules, so intervention will be required. I'd be tempted to rename robots.txt to something else, and provide a route for that path instead which builds the list dynamically. I'd also periodically inspect the renamed robots.txt text file programmatically to get the rules for the dynamic list. I'm going to need this so I might put something together, I'll let you know if I do
    – Clive ♦
    Commented Jan 8, 2019 at 19:56
  • This question is being dicussed on meta
    – Clive ♦
    Commented Jan 9, 2019 at 12:40

1 Answer 1

Reset to default
1

Whatever you are unhappy with, with Drupal's default robots.txt, delete it, and install RobotsTxt to provide an UI to let your SEO person edit the robots.txt according your needs.

Use this module when you are running multiple Drupal sites from a single code base (multisite) and you need a different robots.txt file for each one. This module generates the robots.txt file dynamically and gives you the chance to edit it, on a per-site basis, from the web UI.

Look what I found. This duplicate issue on drupal.org robots.txt: add wildcarded paths for multilingual sites leads to this open issue Fix path matching in robots.txt, which actually could need your input.

But basically some commenters seem to recommend exactly what you came up with, wildcarding the path prefix.

Improve this answer
10
  • 2
    In his comment above, @Clive seems to agree that it is a Drupal-specific question. Good module recommendation, though!
    – hotwebmatter
    Commented Jan 8, 2019 at 22:13
  • I removed the meta commentary about the question, that stuff isn't necessary in answers. We have comments for clarification and meta if it's needed as alternatives. But this is indeed on topic here, how (if at all) Drupal handles a multi-lingual robots.txt file is very much Drupal-specific
    – Clive ♦
    Commented Jan 9, 2019 at 8:00
  • 2
    We don't have a dedicated SEO person :) I would agree this question can be phrased in a Drupal-agnostic way, if other websites have a similar language selection mode based on path prefix. But I think it does have Q/A value to have a Drupal-specific entry point, specifically to state whether or not the shipped robots.txt has this covered or not, and what to do next.
    – donquixote
    Commented Jan 9, 2019 at 10:16
  • 2
    Also I doubt that a user-generated robots.txt is the answer, if that user needs to replicate every rule across languages. This sounds like we would need an automated solution.
    – donquixote
    Commented Jan 9, 2019 at 11:30
  • 2
    The actual question of "What would be the ideal robots.txt for this situation" is independent of Drupal. But the question of how to produce such a robots.txt e.g. dynamically based on language selection settings, this can again have very Drupal-specific answers.
    – donquixote
    Commented Jan 9, 2019 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.