30

One of my Drupal 7 sites has thousands of fields, a bunch of content types, more than 25 views, and hundreds (soon to be thousands) of profile types. Because of this, I'm using a core patch that better caches entity field info (http://drupal.org/node/1040790), and the -dev version of Views which better caches views by display (instead of having one HUGE views cache row with all views data in it).

This has helped most pages on the site to load with 20-30MB of RAM used, rather than 160MB+ (instead of pulling up cache_* table rows for fields and views that were 10MB+, the patches help keep cache_* data much more efficient).

This introduces a problem, however, in that cache rebuilds take a really long time. Usually more than a minute or two. And during this time, Drupal simply won't load any pages (since the caches that it's trying to read from are not yet built, other requests have to wait).

During low-traffic cycles, this isn't a big deal; a hundred or so users will simply have to wait a minute before the page loads. But during high-traffic cycles, the Apache server starts going crazy, with 40+ CPU load, and the memory quickly fills up because all the worker threads sit waiting, and max out their memory, causing swapping. It's kind of a death spiral. A restart of httpd will clear things up, but it takes 5-10 minutes for things to get back to normal.

My goal is to make it so cache clears don't bring the site to its knees. For one, if I use admin_menu's individual cache clearing functions (like "CSS and JS", then "Menu", then "Theme registry", etc.), things go smoothly until I hit the "Page and else" option. That's when views' cache is reset (a very CPU and database-intense operation with the number of views that need to be cached), and when the field info cache is reset (which is also CPU and database-intense on this site).

So... my questions/ideas:

  • Using drush and/or other shell scripting, is it possible for me to clear caches in a more intelligent way than "blast all caches at once, and hope for a clean rebuild"?
  • Can I block http requests while the cache clearing is happening so apache doesn't get clogged with a bunch of cache-stampeding requests?
  • If I can clear caches outside of Drupal/normal httpd request, I could presumably set a higher PHP memory_limit for the cache clear operation, and back off my universal memory_limit (right now set to 256MB, in case any individual httpd thread needs to clear caches...).

Basically: Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?

[Edit for clarification: The main problem I have is cache rebuilds, which (a) take a while, and (b) block all other requests until the rebuilds are complete. I would like to find a way to make it so the rebuilds aren't quite as deadly during high-traffic times.]

  • 2
    Interesting question. If you disable caching, is your site performance adequate? IOW, have you optimized Apache/PHP/MySQL to run as well as it can w/o caching enabled? Obviously, I haven't seen your system, but setting apc.stat=0 and making sure you have enough memory for APC will help reduce disk usage. Using mysqltuner.pl will also give you an indication whether MySQL is the bottleneck. Then you can turn on caching and tweak (it will increase some DB usage, so you may need to adjust MySQL parameters). – mpdonadio Nov 8 '12 at 16:51
  • I use Redis (similar to memcache) to keep the views cache tables in memory. That improved the load times drastically. Looking forward to having the "views cache by display" feature in a stable release, that makes a lot of sense. – uwe Nov 8 '12 at 17:23
  • @MPD - Disabling caching would quickly kill the entire site; typically 100-500 authenticated users, and some sections of the site are pretty heavy. The biggest problem for me is not cache reads (I've experimented with Memcached, Redis, and APC user cache for that), but with the cache rebuild, which is very CPU-intense. – geerlingguy Nov 8 '12 at 21:41
  • Ideally you want to use old cache data while the new cache is being rebuilt. Is this correct? – mikeytown2 Nov 8 '12 at 22:56
  • @mikeytown2 - correct—that would be the ideal. – geerlingguy Nov 9 '12 at 15:49
9

Is there any intelligent and graceful way to clear all caches with Drupal besides simply clicking the button in the UI, or using drush cc all?

The cache actions module does that. It depends on rule. For exampple, you can setup a rule to clear a specific view when a node of type "x" has been added or updated. Checkout the docs for more details.

Also take a look at the cache graceful module - haven't tried it yet but looks interesting.

  • I am already using drush cc [type] for specific cache clearing (similar to cache actions), but I'm more interested in finding ways to clear the cache more gracefully and making sure other httpd threads aren't killing the Apache server. – geerlingguy Nov 8 '12 at 21:45
  • 1
    looks like drush cc will clear all views caches. With cache actions you could just clear a specific view or display. There is probably a bug in the views dev version, otherwise it wouldn't take a minute or two to rebuilt the caches. Do you have the same problem using views 7.x-3.5? Also take a look at drupal.org/project/cache_graceful - haven't tried it yet but looks interesting – uwe Nov 8 '12 at 22:44
  • Views dev breaks out the view displays into their own cache rows, to help with cache read performance. This means that views spends probably 5x more time building the cache (but that helps reduce memory usage when reading caches very much!). – geerlingguy Nov 9 '12 at 15:50
  • Could you add in the information about Cache Graceful into your original answer? I'll accept it, as that particular module helps a bit (but doesn't fix the problem entirely for me). I think I'll have to do a little bit of rearchitecting the site to use fewer fields and entity types to truly fix my problem. – geerlingguy Nov 16 '12 at 18:34
  • ok. I would be interested to hear about your experience with cache_graceful. What part didn't it fix? – uwe Nov 17 '12 at 2:18
2

The main problem is that you are using MySQL to store cache data - for high-load sites this is very ineffective solution.

I advice to use Memcache instead. This will dramatically increase performance of cache system and give you 2 great benefits:

  1. Memcache is much faster for read and write operations that MySQL - all you cache operations (and full cache rebuild) will work faster.
  2. Because cache data not stored in DB anymore - clearing cache will not block any other MySQL queries.

Here is example of Memcache configuration for Drupal 7.

  • I've used memcached and APC both, in various ways, and while they do greatly assist with cache reads, the main problem I have is the actual rebuild; the database is doing almost nothing while the web server is stampeding the cache during the (very slow/long) rebuild process. – geerlingguy Nov 14 '12 at 19:59
  • APC and Memcached make different things. I think that correct configuration of Memcached will help you. BTW, if you site is mostly visited by anonymous users - you can use Varnish. In this case Varnish will use its own cache system and Apache will not be executed for anonymous requests. – Eugene Fidelin Nov 15 '12 at 9:58
  • The site has almost 100% authenticated traffic, otherwise I'd consider using Varnish. I might look into the Cache Graceful module at this point. – geerlingguy Nov 15 '12 at 13:49
0

Using drush and/or other shell scripting, is it possible for me to clear caches in a more intelligent way than "blast all caches at once, and hope for a clean rebuild"?

If you don't want to blast all caches, use: drush cc type_of_cache to clear specific one, or define your own.

Alternatively clear all cache-like tables manually, e.g.

echo "SHOW TABLES LIKE 'cache%'" | $(drush sql-connect) | tail -n +2 | xargs -L1 -I% echo "DELETE FROM %;" | $(drush sql-connect) -v 

If you're using memcached (Bash syntax), try:

pgrep memcached && echo flush_all > /dev/tcp/127.0.0.1/11211

Can I block http requests while the cache clearing is happening so apache doesn't get clogged with a bunch of cache-stampeding requests?

Enable maintenance mode (drush -y vset maintenance_mode 1) to prevent people accessing the site. Or configure front-end to redirect somewhere else (e.g. in Varnish, redirect in Apache or changing .htaccess).

If I can clear caches outside of Drupal/normal httpd request, I could presumably set a higher PHP memory_limit for the cache clear operation, and back off my universal memory_limit (right now set to 256MB, in case any individual httpd thread needs to clear caches...).

Clearing cache doesn't take more memory, but the rebuilding the cache after clear will take more. You can always warm up the caches by running cron or opening any page, e.g.

time php -n -d memory_limit=-1 time $(which drush) cc registry
PHP_OPTIONS='-d memory_limit="2G"' drush cron
php -d memory_limit=1G ./scripts/drupal.sh http://localhost/

Specify -n to ignore php.ini processing which can additionally speed up the cache clearing process.

-1

There's potentially a monetary cost involved, but you could use a caching server setup like Varnish. The upside is that Varnish will serve your site while your cache is clearing on the production server, without the user being the wiser.

The downside: depending on how many seconds/minutes of downtime of the production server vs your VCL timeout settings, Varnish may update during that time and you'll see a Varnish 503 error screen.

But this approach along with Redis or Memcache may help.

  • This question pertains only to internal Drupal caches; the rebuilding of Drupal's caches took forever, and additional layers of caching outside of/in front of Drupal wouldn't do much to help the actual cache data rebuilding (besides offloading some traffic that the webserver would otherwise need to hold for a bit while caches are rebuilt). – geerlingguy Jul 8 '15 at 20:33
  • In that case, I found Zend OpCache to work nicely. :-) – mulderjoe Oct 3 '15 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.