Real world experience in scaling and tuning performance

Question

The website I'm working is allegedly going to have a massive hit rate soon after launch. The client is talking about the possibility of around 2500 hits per second over a day or so.

Ignoring the fact that this hit rate is probably wild client optimism and apart from getting the largest servers possible, what is the best way that Drupal should be configured to support a large hit rate.

I've read Scaling the drupal.org Infrastructure, Drupal performance blog, Best Practices for Scaling Drupal and many other pages, but what I'm looking for is real experience of doing this, what works, what doesn't, and what to expect.

Answer 1

Markdorison's answer is basically the accepted method of attacking this problem. I'll take that a little further.

When you have Pressflow for D6 or Drupal for D7, Memcached and Varnish all working nicely together you'll need to custom code your VCL file. There are free ones available that make starting points but you always need to play with them.

To get Varnish to work optimally make sure you start it with -s malloc xG rather than the default of -s file /path/to/file. Also with Varnish have Varnish cache static items for as long as you can.

If you have more than one web server remove the ETag from the header sent to Varnish in VCL. I also remove Expires and simply rely on Age and max-age in the headers so get browsers back to the site.

Version 1.5 (as of 3rd March 2011) is still the fastest version of Memcached module from Drupal.org. I typically deploy it using a single bin per server to lower tcp traffic for connections to multiple bins at large scale)

Configure the caching in "Performance" to external and set a max age which will send the correct headers to a caching proxy such as Varnish.

If you can't get certain pages to cache properly in Varnish check out blog posts on the web that detail how to inspect the requests. Here is an example post I wrote a while back: What is stopping Varnish and Drupal Pressflow from caching anonymous users page views

You should pick InnoDB (or one of it's other names from other providers like XtraDB) for MySQL and move all tables into it. Then check out this blog post for basic tuning advice http://www.mysqlperformanceblog.com/2007/11/01/innodb-performance-optimization-basics/

Having a large buffer pool is fundamentally important. When load testing the site turn on the slow query log. You probably want to at first capture queries taking longer than 50msec then tune the queries and repetitively reduce the slow log capture time down until you have most queries running using indexes, and executing fairly quickly.

Other basics involve having APC in for PHP. If you go for fast CGI rather than mod_php do spend some time trying to make the APC cache shared across the php instances by configuring a good wrapper script. Also make sure that the APC cache is in a memory mapped file to squeeze every last bit out of PHP.

Answer 2

I would recommend starting with Pressflow (if using Drupal 6), Memcache, Varnish, and some form of Content Distribution Network (CDN) such as Akamai. The end result should be as few of those users as possible actually hitting your origin server.

If you have parts of the page that you are not able to cache for non-anonymous users (things that are specific to that user, "Welcome userX" etc.), you can explore options to populate these pieces of the page such as asynchronous callbacks or edge side includes.

If you have a smaller group of internal users (such as a group of editors) that need to be able to view an uncached version of the site, I would recommend exposing an uncached version of your site at a different URL (protected behind a VPN or equivalent if possible).

Answer 3

• Server side

  • Install Varnish for caching pages for anonymous users.
  • Install a persistent cache system (Memcached, APC, Memcache).
  • Use a CDN such as Akamai to serve static files (JavaScript, CSS, images).

• Code Side

  • Use Pressflow, it allows Varnish to serve cached page for anonymous users.
  • Clean Drupal's watchdog table. Every time a watchdog error gets logged, it consumes CPU resources on the web server and database server. It also increases load time significantly.
  • Implement static and persistent cache strategies until the slow query log comes up clean.
  • Avoid PHP errors that occur within nested foreach loops at all costs.
  • Uninstall unused modules.
  • Turn on caching for Drupal core blocks and Views.

• Database

  • Make sure the tables are properly indexed for faster searching.
  • Do not store unnecessary records, a 100 node database will be always accessed faster than a 3 million node database.

Answer 4

I would also listen to this Lullabot podcast about the way they set up the Grammys.com website for a traffic explosion over the course of a week. It was a pretty educational explanation.

http://www.lullabot.com/podcasts/podcast-92-grammycom

Answer 5

While it is very difficult to predict patterns, if you have a fair idea of the traffic levels. Load test your solution. There are a host of different options and a lot will not be possible to predict until you have live traffic, but if you load test as much as possible at least you will have a fair degree of confidence that your setup can handle traffic.

All the tuning in the world won't help if you don't test it first.

This is was a presentation at DC SF about how the economist did it. http://sf2010.drupal.org/conference/sessions/performance-testing-economist-online-using-grinder

Answer 6

For high-traffic websites you should use multiple servers and load balancer or use simply CDN. Also it's very important to cache as much as possible to minimise load on the web servers.

Using Content Delivery Network (CDN) helps to spread out the resources over several domains (domain sharding) which reduces load on the web server.

Using CDN helps with distributed caching and remote acceleration, also helps to mitigate DDoS attacks, because of multiple end-points. It helps with security, because cached content is more more difficult to exploit.

Example providers: Fastly, Rackspace, Akamai, Azure, CloudFlare, Amazon, MaxCDN, Verizon.

Here are some more suggestions:

• With CDN, use cookieless domains for static components to be cached (like sstatic.net). Since some proxies might refuse to cache the components that are requested with cookies.
• Warm your caches after clearing caches (using wget, Cache Warmer, Drush ECL).
• Use performance monitoring (e.g. New Relic or Yottaa which have integration for Drupal).
• Use monitoring tool for your website (e.g. Nagios).
• Install Varnish and Varnish HTTP Accelerator Integration module, then configure it.
• Varnish + Authcache: Check this Example VCL for Authcache Varnish config file.
• Consider Pound or NGINX in front of Varnish. See: Why Pound is awesome in front of Varnish.
• NGINX can work as reverse proxy and load balancer, so it can replace Pound and Varnish.
• Consider a commercial version of Varnish or NGINX to utilise features not available in the "community" open source version.
• Consider hardware loadbalancer/caching to replace Varnish and Pound (e.g. BIG-IP F5).
• Use tools like ab, JMeter for TTFB, load and stress testing on your web application.

So your web architecture from the user point of view can look like:

1. User (local browser caching).
2. NGINX or Pound + Varnish (load balancer, reverse proxy as HTTP accelerator).
3. Apache (web server).
4. PHP-FPM (PHP FastCGI Process Manager).
5. MariaDB (database).

---

For Drupal optimization suggestion, check: How do you improve Drupal performance?

Answer 7

Enable two extensions :

• Zend OPcache
• wincache

Your performance will work better.

If you are looking to twig the Zend OPcache and Wincache in Microsoft Azure, at first create a folder name ‘ini’ under‘D:\home\site\’. Also, create 2 files, ‘.user.ini’ and ‘settings.ini’

Add following configuration in each file :

.user.ini

[PHP]
post_max_size = 32M
memory_limit = 512M
zend.enable_gc = On
upload_max_filesize = 32M
opcache.enable=1

setting.ini

wincache.ocenabled = 1
wincache.ocachesize = 255

Also, add an App Setting to your Web App with the key PHP_INI_SCAN_DIR and value d:\home\site\ini

After changing PHP_INI_SYSTEM restart your web app. If want to know more about twigging configuration, please check Microsoft documentation.

After above setting, my Drupal (Drupal 8.3) site load within 3 seconds.

Answer 8

You could also examine redistributing the load over multiple servers with the aide of a DNS based or software/hardware load balancing solution. This would also bake in fault tolerance.