We recently upgraded to Drupal 9.3.0 with the composer layout. Each time we execute composer update the contents of robots.txt is overlayed with the file located in core/assets/scaffold/files/robots.txt. We would like to append the default robots.txt with project unique data.

I included the following in composer.json per the documentation I found:

    "extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            "file-mapping": {
               "[web-root]/robots.txt": {
               "append": "assets/robots.txt.append"

I hesitate to create the file core/assets/scaffold/files/robots.txt.append as we have been trained for sometime to not change/add any files in the core/ directory. Where should robots.txt.append be placed to make this work?

1 Answer 1


The path is relative to your project root (where composer.json lives), so in your example create a folder called /assets in the root, i.e. a sibling of /web.

If your question is more whether there's a particular path you should place it at, then the answer is: not especially. The file isn't part of the web app itself, so common sense might say don't put it in /web, but really it's up to you.

Just make sure it's placed somewhere that won't be overwritten during composer updates, and that it's included with your git commits (so not sites/default/files etc), and you'll be fine.

  • I created the file at assets/robots.txt.append where ``assets/``` is a sibling with web/. All is working well! Thanks
    – Greg Sims
    Jan 15 at 20:11
  • 2
    You can also not scaffold it and add the robots txt module.
    – Kevin
    Jan 15 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.