Deploying code to multiple front end servers

Question

We have a situation where we have multiple load balanced servers pointing to a common database.

In the normal run of things this works fine and gives redundancy, scalability etc.

However we are finding deployment to be a bit of a chore.

We are using features extensively and are trying to automate deployment. Deployment at the moment is not very reliable.

In a simplified form the deploy script for each server has two stages

1. Update Files

2. Revert Features (which will manage dependencies, settings changes etc)

Are there any best practices for deploying across multiple servers without causing them to be in an inconsistent state?

As far as I can see if you deploy steps 1 and 2 to server A it will cause server B to break, if you try step one on both servers before proceeding to step 2 they will both be broken for a while.

Answer 1

You should probably look into Aegir, which is by far the most flexible and powerful Drupal site management/deployment system. It is able to manage 'platforms' that span acros multiple web heads, or 'clusters', as well as a variety of other configurations.

I'd suggest having a read through of their community documentation which is generally pretty good, as well as reading through mig5's excellent overview and introductory article and some of his other articles like this one.

You can also get good support in the #aegir IRC channel.

It'll be a bit of a workflow change, which can definitely take some time to get your head around, but once you're there you won't look back.

Answer 2

At our company we maintain A LOT of Drupal sites, our current setup goes something like this:

• Every site's codebase has it's own git repo
• New features not likely to be stable enough for the next release get their own feature branch in git

The above I would say is fairly common for most Drupal sites.

What we do special at our company is debian package the sites for deployment using a custom drush command - 'Drush Debian Packaging'.

Drush Debian Packaging provides a Drush command for building Debian packages of Drupal sites as a means of deploying Drupal sites to Debian or Ubuntu servers.

Drush Debian Packaging utilizes the Drupal hooks system to build a Debian package that best fits your sites needs. Features include:

• Automatic virtual host configuration for Apache2 and Nginx webservers
• Cron setup in /etc/cron.d
• Automated code deployment with partition split for sites/default/files
• Automated configuration through the dpkg debconf settings tool
• Automated deployment process
• custom Git VCS handler for building packages from Git

What does this mean?

To build a release:

cd /path/to/drupal-root
drush dh-make

To deploy a release, first SCP the .deb to all web servers in the cluster. Then on all web servers run (you can use the linux package cssh to type the command to all servers in the farm at the same time):

sudo dpkg -i drupal-site-yoursitehere.2011.05.25-1.all.deb

On one web server run:

cd /path/to/drupal-root
sudo -u drupal-site-yoursitehere drush updb && drush fra -y && drush cron

Done

Of course to roll back this is now trivial from a codebase point of view, simply install the previous version of the .deb to all web servers and rollback the database.

Happy to answer any questions about this

Answer 3

Which part of the deployment process is a chore/unreliable?

If it's the "update server A then B/inconsistency" problem, what about putting up the maintenance page for the duration of your pushes? Maintenance page up, update code on both web heads, run update.php on one of them, maintenance page down. That's pretty easily scriptable.

Another option: depending on the kind of site you run, you could create a "read-only mode" that kicks all users offline and disables login/register. Clone your DB to a second DB on the same db box, clone your front end to a new docroot, do your updates there, then symlink Apache's docroot to the new front end docroot. Workflow is something like:

1. Read only mode on
2. SELECT current_db INTO new_db
3. cp -R current_docroot new_docroot
4. new.yourdomain.com ==> /new_docroot
5. update code into new_docroot
6. update.php
7. symlink /new_docroot ==> current_docroot
8. Read-only mode off.

Answer 4

It will depend on the changes you're making, as Entendu suggests. What proportion of code updates can be run without causing errors if the database isn't updated yet? For anything that's not dependent on database updates (and maybe you can change the development process a bit to make this more common) there isn't really anything special to do. I assume you want to do deployments with minimal downtime, otherwise this would just take some basic sync operations. In this case there will always be some window of time with unwanted effects (even if it's just having the site in read-only mode) but I would think it could be fairly small most of the time.

You can do basic optimization like setting up a "new" directory on each server ahead of time and then switching them all over to point to the new directory at the same time (maybe using symlinks like in Entendu's answer) so you can get all the servers switched to the new files within 5-10 seconds.

That leaves the issue of database updates. If they're the kind that has to be done from one server only, you might want to put the others in maintenance mode or adjust the load balancer to not use them while this happens. Of course if they can't be done while users are active on the site you'll just need to have everything in maintenance mode but for simple updates this might be something you can do in about 30 seconds or less.

It may be worth having different deployment scripts for different types of changes, so you can run the minimal process needed whether that's just copying files, running a small database update, or doing a major database change.

If you can optimize your file and database updates and look at whether there are simple changes you can make to the way things are developed, that could take you closer but I don't know if any of this is new to you :)

Answer 5

Aegir is useful managing a network of sites. I used it to deploy and manage over 2000 sites for a single client.

Your question suggests that you want to manage a single site with multiple webheads. If that is the case Aegir may be less useful for you. Instead I'd suggest you look at using a file system that supports networking. Not only does this ensure your code is kept in sync it means your uploads are available on all nodes.

Historically people have used NFS which allows the file system of one server to be shared with other nodes. Unfortunately this introduces a single point of failure, because if the NFS server locks up or dies your site can't be served.

If you are willing to compromise a little on io performance in favour of a more reliable server, then I would recommend GlusterFS. I have used in a few production environments. It isn't perfect but it is better than NFS. Gluster allows the webhead to always read locally and the writes are then replicated to the other nodes.

In terms of your deployment strategy you should have drush as the first tool on your list. With drush you should be able to automate the steps of your deployment. You should consider adding Jenkins to the mix so you can track your deployment jobs and identify patterns if things fail. Capistrano can be useful for automating the steps involved in the deployment. If you do things properly you can make it so your users never even know that you did a deployment.