When I do a minor upgrade of the core, my custom
robots.txt are overwritten.
How do I prevent that?
You can build a site totally with git to help manage this process. The process is basically
Git clone into your DOCROOT:
git clone --branch 7.x http://git.drupal.org/project/drupal.git /var/www/mysite/docroot
Rename the remote:
git remote rename origin drupal
Checkout the latest tag, and make a site branch for your site:
git checkout 7.26
git branch -b mysite
Now, you can build your site and customize robots.txt and .htaccess to your needs.
When there is a core update, you can:
git fetch drupal
and then use the normal git methods for merging your changes in, eg:
git merge 7.27
or you can preview what will happen, check for conflicts, etc.
If you do this, you need to make sure you don't do a plain:
drush pm-update drupal
as this will then mess things up. I do not think that
drush pm-update --lock=drupal works to prevent this.
One way to solve this, is as follows:
robots.txt, and other files where you're have added customization).
You can script this, using drush for doing the upgrade and git for managing patches and checking for conflicts. This usually makes upgrading quick and painless, even if you've have customized
robots.txt, and other files.
For Apache & configurations without Git, you can alias the root
robots.txt to another file. For example, in
Alias /robots.txt /var/www/html/mydevsite/robots.disallowed.txt
robots.txt will still be overwritten with each core update, but will be unused by visitors & crawlers.