I'm investigating how a Drupal site can be scaled across multiple geographical locations. The primary requirement for this is high availability. The site should be able to run despite some sort of outage on a single location.
The site is a "heavy read, low write" site, meaning that users will never create accounts and edit content. All content is edited by moderators only.
The following two options pop into my mind. I'd be interested in hearing if anyone else have experiences with similar setups - or just general feedback on how viable these options are.
1 WAN-wide cluster
A WAN-wide cluster could be achieved using a database cluster and a storage cluster. However, the immediate implications - as far as I see it - is latency. If we choose to go with a MariaDB/Galera cluster (which is synchronous replication), performance will be very dependent on network latency. For storage, something like GlusterFS (separate storage tier) could solve our needs. However, this again is latency dependent and I'm not so sure replication across an entire WAN is a good idea. The idea of a synchronous cluster (Galera/GlusterFS) is compelling as it removes a few timing issues (replication lag) which you need to take into account otherwise.
The other option for a WAN-wide cluster would be a "regular" master-slave database cluster, where we would have a single master database where all content editing is made. The master database will be located in one location, and the other geographical location(s) will have slave databases only. However, you can never guarantee that some Drupal node not will attempt to write to a database - so all Drupal nodes regardless of location will need to have the possibility to get in touch with the master database to execute write queries. This could be achieved using the AutoSlave module. However, this design makes the other geographical locations dependent on the single master node. If some sort of outage happens here, the master node will no longer be available and all other sites will possibly fail. This defeats the entire idea of high availability. With regards to storage, something simple like lsyncd/csync2 or BitTorrent sync could solve file replication. This would be asynchronous file replication, but I don't think that would be a problem as the site is "heavy read, low write". Only a couple of images a day gets uploaded. Cached JS/CSS and thumbnails would possible be excluded from synchronization, which would put a little more load on all frontends - but that would be a question of testing how much load that would be.
2 Content sharing between locations
I'm not sure how this one could be done, and I'm afraid complexity grows. However, separate - but identical - Drupal installations could be set up at multiple locations. All content editing could be made at one of the sites (master), which would publish content to the other sites. There would be no database or storage dependencies between the sites. However, I'm not sure if this is all that easy to to - especially when content gets complex with multiple fields, media types, images etc.