Performance Problem: Delay on first request

Question

I put together a D7 site with a Minelli subtheme. Along the way I experimented a lot with different themes, different modules. Somewhere along the way I developed an odd performance issue, and now I don't really know what theme/module/config caused it.

The problem is that when I first visit the site, it takes about 15 seconds before the first page shows. I can then move around the site and it is very responsive. If I leave it for an hour or so, then come back to it, the first request is again very slow.

I have cleared the cache's so that shouldn't be the problem. Also, I have disabled themes and modules which I am not using. I moved the site to new infrastructure but the problem followed it!

Where do I go next?

Answer 1

There are three things that I would check.

One, if you are on a production site and not editing PHP files, then you should ensure that APC is enabled, has enough memory, and has a long TTL (you could go with a day or never expire if you wanted). You can also consider setting apc.stat=0. The APC docs have all of the information you need for setting the TTL. For choosing the amount of memory, you should stick the apc.php file somewhere protected and monitor the memory usage and churn statistics. Adjust the APC memory so that your miss rate is very low. Initial slowness could be because APC is full and emptying out (IIRC, APC dumps the entire cache when it is full instead of employing LRU or more advanced cache strategies).

Second, make sure you have MySQL tuned appropriately. You can use mysqltuner to adjust your buffer sizes. Your initial slowness could be because of loading tables from disk and/or query cache misses. While not perfect, mysqltuner does get you going in the right direction.

Third, make sure you have a real Drupal cron strategy. Personally, I would disable automagic cron on "admin/config/system/cron" and set up a crontab to run every night. You can also try out Elysia Cron if you really do need finer grained control over things. This way you can the necessary tasks run as often as you need, but have the normal ones run overnight. Your initial slowness could be from cron runs happening every hour. You can confirm this by looking at when cron runs on "admin/reports/dblog" and trying to match up with your slowness.

Answer 2

Ivanhoe123 is probably right: Drupal 7 comes with 'poor mans cron' enabled by default. In short, it means that (once in a while) cron is run before Drupal renders the page, delaying everything.

Always try to use a real cron job on production sites. For more technical details see http://drupal.org/cron, or talk to your hosting company.

To disable it, go to admin/config/system/cron and select 'Never'.

Answer 3

The Devel module offers database logging to check whether you have any long running queries.

If this doesn't help take either XHProf or XDebug and find the guilty code. XHProf (a profiler) draws you a nice map of all the functions that are being executed on the server, and it tells you which ones are consuming the most execution time. On the other hand, when XDebug (a debugger) is configured with an IDE such as Eclipse (see video) it allows you drill down into every function being executed LIVE. The profiler will give you an idea of what is being executed; while the debugger will show you why is it being executed.

Answer 4

Just from the flavor of the question, I immediately think of three(3) things

• MySQL Storage Engine/CPU
• Database Caching
• Table Locking

MySQL Storage Engine

If you are not using any FULLTEXT searching/indexing, I strongly recommend you convert all your MyISAM data into InnoDB. MyISAM is not designed to take advantage of multiple CPUs and multiple cores. InnoDB has been greatly enhanced for multiple CPU usage as well as read/write hyperthreading.

Here are some posts I made about this in the DBA StackExchange and in this site with regard to tuning MySQL for InnoDB Performance

• Sep 20, 2011 : Multi cores and MySQL Performance
• Sep 12, 2011 : Possible to make MySQL use more then one core
• Jun 07, 2011 : How do I convert a database from MyISAM to InnoDB?
• May 26, 2011 : About single threaded versus multithreaded databases performance
• Apr 15, 2011 : https://drupal.stackexchange.com/questions/1715/what-would-be-the-optimal-mysql-configuration-for-a-drupal-7-site/2367#2367

Database Caching

Another strong argument for converting all MyISAM data to InnoDB is how MySQL caches data/indexes. The MyISAM Storage Engine only caches indexes. InnoDB caches data and indexes. In light of this, you can allocate enough memory for the InnoDB Buffer Pool to accommodate one of the following (whichever is smaller)

• All InnoDB Data and Indexes (Ideal if you have enough RAM for it as well for the OS; eliminates subsequent delays)
• 75% of Installed RAM (in case you have more InnoDB data/indexes that RAM ; minimizes delays)

If you are using MySQL 5.1, you could set innodb_max_dirty_pages_pct = 0. This will slightly increase disk I/O, but the InnoDB Buffer Pool will be cleared enough to allow old data and index pages to rotate out without disk I/O surges. MySQL 5.5 and MySQL 5.1's InnoDB Plugin do not need this adjustment since it has a better default Buffer Pool flushing mechanism.

If using InnoDB is out of the question, you may need to go with memcached or varnish. This allows the developer to dictate how long cached data will reside in server RAM. Naturally, this will require developmental enhancement to make your application memcached/varnish-aware.

Table Locking

• InnoDB does row-level locking, which allows for transactions and nonblocking reads.
• MyISAM only does full table locking for every INSERT, UPDATE, and DELETE.

Epilogue

You cannot avoid an initial delay after a MySQL Restart. Yet, once you enhance MySQL using the aforementioned suggestions/information, you should no longer experience subsequent delays.

Answer 5

I would use tools like YSlow or Firebug etc. to determine exactly what is happening when you load said page and when you load said page immediately after. Check if it's a caching issue and furthermore, check how it functions when you access the page as an anonymous user and then as an authenticated user. Compare this to your performance settings within Drupal.

If it's not a caching issue, then use Devel's query log as well as MySQL's logs to see what's happening w.r.t. the database. Additionally, if you have opcode or similar performance-enhancing caches on the server, try getting some numbers with them off and then back on.

Answer 6

Sounds like the cron is running.

Check your settings here: admin/config/system/cron

Answer 7

I nearly dropped Drupal for my last project because of this.

There must me more than one cause though. I have yet to find a 'fix all' solution that works every time this issue creeps up.

Syslog and Ubuntu/Debain

The first time I ran across the intermittent 15 second load time was while running drupal on (Dedicated, non-shared) Debian/Ubuntu based systems. Disabling the Syslog module was the solution for me.

As @BetaRide said, using xDebug or some other PHP profiler is extremely enlightening.

---

Still A Problem - A Workaround

As for my other installs, I'm still at a loss.

This issue is more noticeable on my development server and my low traffic Drupal installs.

As a workaround I've setup a cron job to load the sites home page every 60 seconds as well as Drupal's cron script every 300 seconds. This is obviously not optimal, but I would rather wget or curl experience the 15 second load time instead of a human visitor.

Answer 8

Many people suggest this issue could be related to blocking synchronous background processes, particularly related to heavy cron jobs.

If true, there exists a great pair of modules under active development by gielfeldt* that might cut this issue right out, or at least, could offer some clues and help site builders diagnose and treat specific culprits in their cases. Both replace blocking synchonous processes with non-blocking asynchronous HTTP or commands, and both offer relevant reports that can identify troublesome processes:

• Background process and its bundled modules allow for Drupal's background processes queue to be processed asynchronously, so they don't block. This could stop the problem. Also, with the bundled Background Process Apache Server module in the latest dev, there is a basic but improving UI report with features to oversee, unlock and inspect start times and progress of these processes. This could identify the problem process.
• Ultimate Cron builds on Background Process to allow cron-triggered tasks to have their own seperate asynchronous scehdules, each of which can be monitored and stopped in a UI. As well as being great for seperating heavy duty performance-sapping tasks from regular low-overhead cleanup, it also gives you a report with convenient information such as the running duration of each individual cron-triggered task, when they last run, current status, etc. This could also remove the blocking from, and/or identify, problem processes.

Both are very useful modules anyway; for this problem, they can be used to test the (very plausible sounding) theory that the blockages are caused by synchronous blocking processes or cron runs. Potentially, they could solve the issue by running these asynchronously instead of synchronously, and they could also potentially offer clues as to which specific processes were causing the hold up. (be warned that their documentation is very much a work in progress...

If, however, they can't be configured to help at all, that suggests there's more to the problem than just synchronous background processes. FWIW, I've never had this particular issue on a site since getting these modules to work properly (yet - touch wood) - but I have seen it on my sites before, as well as on live Drupal sites in the wild.

Also be aware of other related plug-in modules currently in development - e.g. in complex high-intensity cases, Ultimate Cron Queue Scaler, which allows threshold-based throttling, might help reduce cron-related performance problems.

---

*no affiliation, I'm just a highly impressed user of their work

Answer 9

Since this is hitting me once more I startet do investigate the problem. I can definitely confirm that

1. a call to drupal_cron_run() triggered by poorman's cron of core adds ~5s to the request time on my dev machine. This can be tested by uncommenting the tests around the call to drupal_cron_run() in modules/system/system.module in system_run_automated_cron()
2. clearing all caches adds ~2s to the request time on my dev machine. This can be tested by doing a drush cc all and reloading the page again.

This means that setting cron to never and adding a call to cron via crontab makes the situation a lot better. Hitting some often used pages right afterwards to refill the cache would again improove the user experience.

I'm not sure though about the caching. I haven't touched the default cache settings for this site. I think that drupal is rebuilding all caches from time to time perhaps triggered by cron, but I'm not sure how this is done. But a 7s delay is pretty much what I see when I hit the page after some hours.

Answer 10

Issues like this can drive you nuts and when I had been in similar situations helps to figure out what is causing the issue going one step at the time and then test it as an anonymous and logged user. (onion layer method)

You mention you start noticing the issue after playing with a couple of theme and custom coding your own. I don't know how complex is your site nor the logic behind it, but the following steps will help you find the issue:

1. In your server create a folder or another account (this might be better) where you are going to do a clean Drupal install with the same version your are using on your site. Then without adding any module or theme test the time it take for the site to respond the first request and the following request. If all works well you can ignore server configurations issues, if it is behaving the same as your current the you have a configuration error either with your web server or database.

2. If the results from step 1 are good and the server is responding fast and following requests are as fast, then install only the theme from your current site into the clean install site and test it again. If everything still respond fast, then your theme is not the problem and you should continue to step 3 otherwise you have to start debugging your theme *1.

3. If after the tests on step 2 the site still fast start bringing over the modules in your current site and make sure to test the response time after adding and enabling each module *2.

4. If after adding the theme and modules the site still responding fast start adding the configuration, create content types, import views, setup menus, etc. Don't forget to test the site response after adding each one.

5. Setup and configuration ready and the site still fast, well now bring the data over. Import nodes, taxonomy terms, comments, etc. I know I must sound like a broken record, but always test after completing each step.

*1 Testing themes: this process can be tricky in a super elaborated theme, here are a couple of pointers:

1. If you link to any external js or css library, try to use a local copy of the same.

2. In your template.php file check for function that might have longer or endless loops as well as preprocess function and/or hook theme functions.

3. Check other template file (page.tpl.php, etc) and look for raw PHP processing of arrays and objects.

4. If using "Views" and views template files, then check then as well.

5. Always double check the paths, optimize images, js and css files. Sometimes js files can have some heavy height when use multiple snippets of code in a single file.

*2 Testing modules: testing modules is a little different because the use of heavy manipulation with PHP is allowed. Here are some pointers:

1. Community supported modules (CCK, Views, etc) have issues queue on drupal.org check them to see if there is any existing issue about your problem and if there is chances are that there might be a patch to fix it.

2. Own custom coded module, well if you coded you have to fix it, right?. Double check your coding and check the functions usage against api.drupal.org, you might be using an overkilling function instead of a hook.

3. Internet shared custom code module, do as in step 2, but this time you can also contact the original module writer and let him/her know about the problem.

4. If your site is an upgrade ( D5 -> D6 -> D7 ) check the migration or upgrade scripts (usually in module.install file) you might need an extra "index" on the new table configuration to make the slow SQL query X faster.

5. If you feel you have tunnel vision about the problem, step out for a bit and do some other activity completely un-related and then come back later to revisit the issue.

6. If you ping point the issue on a section of code, but can not make heads or tails about how to fix it, try explaining what that section is suppost to do to a person that has no idea how to program or how Drupal work and be ready to be surprise.

Note: Don't be alarm if after re-building your site all start to work like a charm that is one of best features computers have.

Answer 11

Double check that you haven't deleted any modules without uninstalling them. This causes a delay because Drupal tries to find the files but they aren't there anymore.

Delete references in the variables table if the modules aren't existing anymore.

Answer 12

A Web APM such as newrelic is the best tool for tracking down performance issues. I've had sites calling one or two lines of code that did wonky things, loaded unnecessary arrays at weird times, and did other things that were pretty invisible until we tracked them down with an APM.

Answer 13

Someone mentioned that GoDaddy will be slow. Many cloud-based hosting companies will also have this initial delay because services like AWS have it. It's cheaper to have servers deprioritized automatically, and those servers will require a second or two to 'wake up.'

For instance, Pagodabox has 3-4 seconds for first byte, until the server is happily awake. Pagodabox has in fact monetized keeping the server awake, and so you can pay extra to 'caffienate' your site.

Also, a CDN can help you out. Your web/db sever won't be loaded down with serving cached pages or images. A good tutorial here: http://wimleers.com/article/easy-drupal-cdn-integration-for-fun-and-profit

And... WebPageTest makes me happy. http://www.webpagetest.org/ Compare load times around the planet and with different web browsers for free. Use this to get real-world results for whatever changes you're making.

Answer 14

the problem could be anywhere.

1. Make sure you have not turned on debug mode on any theme or module. For example, in many themes there is an option to regenerate theme registry.
2. If you are running on shared hosting like Godaddy, then 15 sec for the first time request is normal.
3. Convert your site or front page to codebase using Drush CTools Export module. This will eliminate any database call and your site will run entirely from php.
4. If you still get problems, use Devel settings by turning on query log and page timer options at admin/config/development/devel. See which of the two takes more time to generate entire page.
5. Restart your server if nothing works.
6. Worst case install XHProf to see where things are going wrong.

Answer 15

So this is how I fixed the problem for my install. It is not a real solution as I could not nail the exact source of the problem (if there is one), but it is a good fix

1) Aggregate CSS (cache settings). This reduced the latency by half

2) Set cron to never (and run it externally) - Note: I had"attempting to start cron while it is already running" errors. I think it was trying to start cron at every launch but since it failed, the cron page was not mentioning the latest attempt, but rather the latest success.

3) Set up a cron job that calls the home page with Lynx every 30 minutes

All this on a shared hosting server. It is not optimal but it works

Answer 16

I'd suggest using a front-end cache along the lines of the Boost module (assuming you're on shared hosting) or Varnish. This will work best if your site accesses are primarily anonymous and page content is, for the most part, non-dynamic (i.e. the pages don't change much).

These solutions save the rendered pages on first access and then serve the pre-rendered html instead of going through the full Drupal bootstrap, page building, and theming processes, saving a LOT of time, especially on busy sites but also on sites such as you describe which "go to sleep" and take too long to wake up.

The only real drawback is that (at least for Boost) you'll need to clear cache when site content changes. If you want to make sure the site is fully cached with current content, you can run drush cc all and then curl or wget against the full site periodically via cron.