Assuming there are no module updates that require doing a database update, if it's just an update to a newer minor release of Drupal core, as well as of contrib and custom modules, what potential pitfalls should I watch out for when updating without entering maintenance mode?
We have 3 load-balanced web servers that all point to the same database and file server. The updates are generally done by first putting all the 3 Drupal instances into maintenance mode, waiting for a minute or two (not sure if this is necessary), then literally overwriting the entire application root directory with the newer version on each server, then clearing the caches on each of the servers, then exiting maintenance mode.
Is it safe to do this overwriting of the files while the site is live, without entering maintenance mode? I've heard of people doing this, but they have brochureware sites with not a lot of traffic and not a lot of logged-in user sessions (which we do have). Does doing so become more or less safe if you have multiple load-balanced web servers, use APC, use Varnish in front of the web servers, use nginx, use Memcache (which also stores sessions) between the web servers and the DB, have all pages using SSL, etc?
It would be great to seamlessly roll over to a new version without having to enter maintenance mode, but it feels risky to me. Are there edge cases where if your new files take a couple of minutes to overwrite the old files in the application root, then maybe some users are getting loading one module from the old set of files and one module from the newly-copied set of files, leading to strange and non-reproducible errors?