On a site of mine drush cc all
takes more than 4 mins to run. The site db is a few GBs. However, I cannot see a clear reason as to why it takes too long. What can I do to locate the bottle neck?
-
Check first mysql query log: drupal.stackexchange.com/questions/75629/…– AgAJan 30, 2014 at 17:02
-
Do you have cron running?– mpdonadio ♦Jan 30, 2014 at 20:14
-
Yes I do have a cron running. The site is slow in general. So many legacy code that is not worth re factoring.– awmJan 30, 2014 at 22:21
-
I agree with @MPD here, I don't think this is memory related. MySQL is also just one of the possible reasons. There's only one way to find out and that's to profile it. The easiest way to do that is using the xhprof extension and devel, that also works with drush (use -d to see the link to the report). My guess is that there are a number of issues, A typical problem if the site is slow for all requests is missing modules, see drupal.stackexchange.com/questions/724/why-is-drupal-7-so-slow. Views also has some major performance issues with caches, see drupal.org/node/1944674.– BerdirFeb 5, 2014 at 8:44
-
Thank you, I figured that I have to do profiling, but in the case of drupal code base, it's always hard to pinpoint the bottleneck. Although I said the site is slow, it is not too slow and drush cc all, is unproportionally slower.– awmFeb 6, 2014 at 17:09
6 Answers
The clearing cache it-self doesn't take a long time, since it's just truncating cache tables. What takes time the most is rebuilding the cache registry after that. Usually it's theme registry which scanning for all new hooks and all your Drupal folders for new template files, system which scanning files for the new modules and class files, and similar.
You can always clear specific cache by specifying drush cc theme-registry
or any other.
It's highly recommended that you use PHP caching mechanism (e.g. OPCache, XCache, etc.) to speed up the processing. And memory-base cache to replace heavy usage on SQL tables (e.g. memcached or redis), so clearing cache would take no time, just by flushing the cache (e.g. echo flush_all > /dev/tcp/127.0.0.1/11211
in Bash).
Alternatively you can always clear cache manually, e.g. by:
echo "SHOW TABLES LIKE 'cache%'" | $(drush sql-connect) | tail -n +2 | xargs -L1 -I% echo "DELETE FROM %;" | $(drush sql-connect) -v
Debugging
To check what's taking the most time specifically, you need to debug/profile it (e.g. XDebug, XHProf, phpdbg, dtrace).
On OS X/Unix, this can be achieved by dtrace
(after running drush
):
sudo dtrace -qn 'php*:::function-entry { printf("%Y: PHP function-entry:\t%s%s%s() in %s:%d\n", walltimestamp, copyinstr(arg3), copyinstr(arg4), copyinstr(arg0), basename(copyinstr(arg1)), (int)arg2); }'
On Linux, use strace
, e.g.
strace -e trace=sendto,recvfrom -s1000 -p $(pgrep php)
To look for some specific things, try adding e.g.: strace ... 2>&1 | grep -C5 UPDATE
.
If you have a really large database, and your system is running fine when you are not doing a cache clear, it is likely that you do not have enough memory to fully support your setup.
If your site is running on a Linux box, run 'top' (from your shell) and press shift-M to sort the process list by memory used. Then, run your cache-clear operation from another terminal. You should see mysql and apache rise to the top of the list. You'll be able to see what percentage of total memory each of these processes are using, and how much free RAM is used. If you have a large amount of virtual space, but all of your physical RAM is exhausted, this operation could be causing VM memory to thrash, which can drive your execution time down to a small fraction of what it normally is.
Once, I was running a mid-traffic Drupal site on a box w/out -quite- enough memory to support the setup. When I ran a cache-clear on an unrelated low-traffic site, the cache rebuild pushed the system past its limit, and everything all but locked up. So, the total system behavior is important here; this is why simple tools like 'top' are a handy place to start.
-
Thanks, I think the underlying cause is that the site has been around for a while, the db is a bit large and the code terribly needs re factoring. For example, a node_delete operation takes about 3 seconds. I think giving too much memory may be redundant, do you agree?– awmJan 30, 2014 at 22:23
-
I have it on my machine and it is also slow. I doubled the memory, up to 1GB and timed it ` time drush cc all`. Not much better only got 4s faster.– awmJan 30, 2014 at 22:38
-
1Yes, if the whole site is slow all the time even with a lot of memory, then cc is not the problem; you'll have to do more general profiling. Jan 30, 2014 at 23:25
-
Also, if the site is really old and needs a refresh, consider rebuilding from scratch with fresh modules, and use drupal-to-drupal migration (drupal.org/project/migrate_d2d) to move your content over. Jan 30, 2014 at 23:27
I am going to disagree (somewhat) with @greg_1_anderson here.
If the system isn't crashing totally during a cc all
, then I don't think you have a general memory issue. When a LAMP server runs out of memory, it will hit swap. An active server hitting swap will cause an avalance of badness. httpd processes will start to stack up due to system slowdown (swap makes a system run very slow), which will cause more swap to be used, etc. On sites where I have seen this happen, I would see the process load hit 100, and a ton of active httpd processes.
If your system eventually comes back, then I think you are poorly tuned. drush cc all
will result in a lot of database accesses, so I think it is showing the problem more. My suggestion would be to run mysqltuner on the site. If you have a multi GB database, my guess is that your innodb_buffer_pool_size
isn't even remotely sized properly, and your MySQL instance is thrashing. I would also investigate an alternate cache backend to try to keep the database footprint smaller.
-
Thanks, drupal is used only for editoria use that has a servie layer which has varnish infornt of it. Users do now get to drupal. I just want to investigate whats happening because I think there is bottleneck. I think my best bet is to turn on the mysql slow log and monitor the db. And then do some profiling with xdebug– awmFeb 1, 2014 at 21:36
-
SHOW FULL PROCESSLIST will tell you what's happening without need to use slow query log (and so without server restart). See other answer for mytop as well. Feb 4, 2014 at 21:30
It could be your web hosting environment. Are you referring to a local setup, or something hosting on shared hosting or a VPS/server?
- hosting environment - if you are on shared web hosting, the amount of memory Drupal/drush can use will be limited see: https://drupal.org/node/207036
- max execution time - needs to be increased
-
drush normally isn't limited by
max_execution_time
. You can dodrush php-eval "print ini_get('max_execution_time');"
to double check, though.– mpdonadio ♦Jan 31, 2014 at 17:16
This is not a complete solution, just one more tool to help identify the source of your delays.
As well as using top
to monitor processes, you might find the output of mytop
informative. (Other answers above presume MySQL but if you're using another DB backend, you'll need to swap mytop for an equivalent tool.)
mytop
simply executes MySQL SHOW FULL PROCESSLIST
in a loop, and shows you which queries are being executed (so which take a lot of time). If the cache clear is taking a long time cleaning up this or that table, you'll see exactly what's holding things up here. If you don't have access to install mytop
, just do a crude version in your shell -
while true; do mysql -e 'SHOW FULL PROCESSLIST' && sleep 5 && clear ; done
If the delay is not stemming from MySQL queries, then this tool may at least confirm that for you.
I found a blog post entitled Speed Up Cache Clearing on Drupal 7 to be very helpful in narrowing down precisely what portion of the cache clear process was slowing things down. The issues it pinpoints in features module and the entity api module were affecting my site, but the process detailed in the post also helped me track down issues we were encountering in Drupal core and breakpoints module.
It takes some time to work through the process, but it has helped me to reduce cache clears down from multiple minutes to under a minute.