The default robots.txt shipped with Drupal contains rules such as this:
If I understand correctly, the leading slash means this really has to match the beginning of the path. E.g. a path like
/xyz/user/register/ would not match the rule. (Please correct me if I'm wrong)
(For the ending slash I am not sure)
With the language path prefix option, Drupal will have urls like
It can be assumed that if we want to exclude /user/register, we also want to exclude /fr/user/register and other language versions.
/*/user/register/ might be too generic, as it also would match other urls which we don't want to exclude.
- Will Drupal's default robots.txt also exclude language-specific versions of the targeted urls? (according to my own assumption it does not).
- What would be the simplest / cleanest way to exclude all language versions for a given path?
I think this applies equally to Drupal 7 and Drupal 8.