Memory-saving cache-clearing strategies for large sites?

Question

One of my Drupal 7 sites has thousands of fields, a bunch of content types, more than 25 views, and hundreds (soon to be thousands) of profile types. Because of this, I'm using a core patch that better caches entity field info (http://drupal.org/node/1040790), and the -dev version of Views which better caches views by display (instead of having one HUGE views cache row with all views data in it).

This has helped most pages on the site to load with 20-30MB of RAM used, rather than 160MB+ (instead of pulling up cache_* table rows for fields and views that were 10MB+, the patches help keep cache_* data much more efficient).

This introduces a problem, however, in that cache rebuilds take a really long time. Usually more than a minute or two. And during this time, Drupal simply won't load any pages (since the caches that it's trying to read from are not yet built, other requests have to wait).

During low-traffic cycles, this isn't a big deal; a hundred or so users will simply have to wait a minute before the page loads. But during high-traffic cycles, the Apache server starts going crazy, with 40+ CPU load, and the memory quickly fills up because all the worker threads sit waiting, and max out their memory, causing swapping. It's kind of a death spiral. A restart of httpd will clear things up, but it takes 5-10 minutes for things to get back to normal.

My goal is to make it so cache clears don't bring the site to its knees. For one, if I use admin_menu's individual cache clearing functions (like "CSS and JS", then "Menu", then "Theme registry", etc.), things go smoothly until I hit the "Page and else" option. That's when views' cache is reset (a very CPU and database-intense operation with the number of views that need to be cached), and when the field info cache is reset (which is also CPU and database-intense on this site).

So... my questions/ideas:

• Using drush and/or other shell scripting, is it possible for me to clear caches in a more intelligent way than "blast all caches at once, and hope for a clean rebuild"?
• Can I block http requests while the cache clearing is happening so apache doesn't get clogged with a bunch of cache-stampeding requests?
• If I can clear caches outside of Drupal/normal httpd request, I could presumably set a higher PHP memory_limit for the cache clear operation, and back off my universal memory_limit (right now set to 256MB, in case any individual httpd thread needs to clear caches...).

Basically: Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?

[Edit for clarification: The main problem I have is cache rebuilds, which (a) take a while, and (b) block all other requests until the rebuilds are complete. I would like to find a way to make it so the rebuilds aren't quite as deadly during high-traffic times.]

Answer 1

Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?

The cache actions module does that. It depends on rule. For exampple, you can setup a rule to clear a specific view when a node of type "x" has been added or updated. Checkout the docs for more details.

Also take a look at the cache graceful module - haven't tried it yet but looks interesting.

Answer 2

The main problem is that you are using MySQL to store cache data - for high-load sites this is very ineffective solution.

I advice to use Memcache instead. This will dramatically increase performance of cache system and give you 2 great benefits:

1. Memcache is much faster for read and write operations that MySQL - all you cache operations (and full cache rebuild) will work faster.
2. Because cache data not stored in DB anymore - clearing cache will not block any other MySQL queries.

Here is example of Memcache configuration for Drupal 7.

Answer 3

Using drush and/or other shell scripting, is it possible for me to clear caches in a more intelligent way than "blast all caches at once, and hope for a clean rebuild"?

If you don't want to blast all caches, use: drush cc type_of_cache to clear specific one, or define your own.

Alternatively clear all cache-like tables manually, e.g.

echo "SHOW TABLES LIKE 'cache%'" | $(drush sql-connect) | tail -n +2 | xargs -L1 -I% echo "DELETE FROM %;" | $(drush sql-connect) -v 

If you're using memcached (Bash syntax), try:

pgrep memcached && echo flush_all > /dev/tcp/127.0.0.1/11211

Can I block http requests while the cache clearing is happening so apache doesn't get clogged with a bunch of cache-stampeding requests?

Enable maintenance mode (drush -y vset maintenance_mode 1) to prevent people accessing the site. Or configure front-end to redirect somewhere else (e.g. in Varnish, redirect in Apache or changing .htaccess).

If I can clear caches outside of Drupal/normal httpd request, I could presumably set a higher PHP memory_limit for the cache clear operation, and back off my universal memory_limit (right now set to 256MB, in case any individual httpd thread needs to clear caches...).

Clearing cache doesn't take more memory, but the rebuilding the cache after clear will take more. You can always warm up the caches by running cron or opening any page, e.g.

time php -n -d memory_limit=-1 time $(which drush) cc registry
PHP_OPTIONS='-d memory_limit="2G"' drush cron
php -d memory_limit=1G ./scripts/drupal.sh http://localhost/

^{Specify -n to ignore php.ini processing which can additionally speed up the cache clearing process.}

Answer 4

There's potentially a monetary cost involved, but you could use a caching server setup like Varnish. The upside is that Varnish will serve your site while your cache is clearing on the production server, without the user being the wiser.

The downside: depending on how many seconds/minutes of downtime of the production server vs your VCL timeout settings, Varnish may update during that time and you'll see a Varnish 503 error screen.

But this approach along with Redis or Memcache may help.