Pitfalls when updating a site without entering maintenance mode?

Question

Assuming there are no module updates that require doing a database update, if it's just an update to a newer minor release of Drupal core, as well as of contrib and custom modules, what potential pitfalls should I watch out for when updating without entering maintenance mode?

We have 3 load-balanced web servers that all point to the same database and file server. The updates are generally done by first putting all the 3 Drupal instances into maintenance mode, waiting for a minute or two (not sure if this is necessary), then literally overwriting the entire application root directory with the newer version on each server, then clearing the caches on each of the servers, then exiting maintenance mode.

Is it safe to do this overwriting of the files while the site is live, without entering maintenance mode? I've heard of people doing this, but they have brochureware sites with not a lot of traffic and not a lot of logged-in user sessions (which we do have). Does doing so become more or less safe if you have multiple load-balanced web servers, use APC, use Varnish in front of the web servers, use nginx, use Memcache (which also stores sessions) between the web servers and the DB, have all pages using SSL, etc?

It would be great to seamlessly roll over to a new version without having to enter maintenance mode, but it feels risky to me. Are there edge cases where if your new files take a couple of minutes to overwrite the old files in the application root, then maybe some users are getting loading one module from the old set of files and one module from the newly-copied set of files, leading to strange and non-reproducible errors?

Answer 1

As you said there:

but they have brochureware sites with not a lot of traffic and not a lot of logged-in user sessions (which we do have)

If accessibility is very important to your website I would suggest you do NOT take the risk to directly overwrite files for upgrading while the website is still up and serving. It may lead to corrupt data or broken response.

One approach could be take one web server out of the load-balance group, upgrade it while your website is still up and serving. After upgrade and test, put it back then upgrade another one, until all are done. In this way you can seamlessly upgrade without shutting down the service. It may sound complicated and boring, but it could be highly automated by writing some scripts I think.

Answer 2

It is risky and should generally not be attempted. Drupal code makes assumptions about the database structure and should generally not be run when the accompanying database updates have not been applied.

Currently, the best you can do is:

• Make sure your site uses a mechanism that symlinks the PHP code in the correct location.
• Copy over all code for the new release to a fresh location
• Enter the site into maintenance mode
• Switch over the symlink to the new code (this is much quicker than copying or even rsyncing over the new code); you are likely to need to restart your web server for it to pick up the new symlink
• Run database updates and hope they do not take a long time
• Disable maintenance mode

For lengthy update processes, you can check if they really need to run during maintenance mode. If you are e.g. just updating content and there is no great damage in running the site without the updates applied everywhere, you can consider stopping the actual update short (maybe patch the update hook to just return TRUE), and devising something to run the update script while the site is live. Of course, you will need to take greater care of the load this would cause and if it doesn't interfere with regular site operations. Unfortunately, Drupal does not contain such a mechanism.

When running something like Varnish, there are definitely ways to minimise the number of occasions where users actually see the maintenance page, e.g. have Varnish serve cached content when it finds the backend is not available. It would be rather out of scope here to go into that very deeply, though, I guess.