The default robots.txt shipped with Drupal contains rules such as this:

Disallow: /user/register/

If I understand correctly, the leading slash means this really has to match the beginning of the path. E.g. a path like /xyz/user/register/ would not match the rule. (Please correct me if I'm wrong)

(For the ending slash I am not sure)

With the language path prefix option, Drupal will have urls like /fr/user/register etc.

It can be assumed that if we want to exclude /user/register, we also want to exclude /fr/user/register and other language versions.

However, a /*/user/register/ might be too generic, as it also would match other urls which we don't want to exclude.

Questions:

• Will Drupal's default robots.txt also exclude language-specific versions of the targeted urls? (according to my own assumption it does not).
• What would be the simplest / cleanest way to exclude all language versions for a given path?

I think this applies equally to Drupal 7 and Drupal 8.