The default robots.txt shipped with Drupal contains rules such as this:

`Disallow: /user/register/`

If I understand correctly, the leading slash means this really has to match the beginning of the path. E.g. a path like `/xyz/user/register/` would not match the rule. (Please correct me if I'm wrong)

(For the ending slash I am not sure)

With the language path prefix option, Drupal will have urls like `/fr/user/register` etc.

It can be assumed that if we want to exclude /user/register, we also want to exclude /fr/user/register and other language versions.

However, a `/*/user/register/` might be too generic, as it also would match other urls which we don't want to exclude.

Questions:

- Will Drupal's default robots.txt also exclude language-specific versions of the targeted urls? (according to my own assumption it does not).
- What would be the simplest / cleanest way to exclude all language versions for a given path?

I think this applies equally to Drupal 7 and Drupal 8.