3

We are in the process of migrating our website to Drupal 7 and we had a failed launch this past week. We performed load testing and were certain that our server was going to work just fine with our current load, unfortunately we did not perform load testing with logged-in users (we use a 3rd party gateway login system and it was almost impossible to accomplish a load test this way). Long story short - logged in users overloaded our site and the server crashed.

So we've gone back to the drawing board, implemented APC and Memcache and we believe we have the problem fixed. The CPU no longer maxes out when under an expected load.

However, we now have a new issue that is a critical showstopper if we cannot find a fix for it.

While the server is under load from users, if an administrator attempts to enable or disable a module, ANY MODULE, (and sometimes even if we attempt to VIEW the modules page), the CPU will max out and the server will then eventually crash. The only option at that point is to restart the httpd service, which of course disrupts all user traffic.

When operating normally, CPU usage is at about 20 - 30%. When you attempt to enable or disable a module, the server IMMEDIATELY pegs the CPU at 100% and becomes nearly unresponsive from the browser and command line. Our normal page load times are 400ms, and they skyrocket to over a minute until the server crashes.

With no user load, the server operates as expected, though the action of enabling / disabling a module does spike the CPU for a short period of time. I have checked the sql slow query log and haven't found anything enlightening there. HTTPD is not spitting out any errors either.

Server has 2 CPU, 16GB memory and is dedicated to running this site alone; surely it isn't under powered.

What could possibly be causing this problem?

8

If you look at the source for module_enable() you can see what is going on. In particular, several cache tables get cleared, as does the registry. It also does filesystem scanning to rebuild module data. All of these are expensive operations.

If you look at the source for system_modules() (this is the callback for the modules page), you will see that system_rebuild_module_data() is called. This is also an expensive operation.

My guess is that something on your server isn't optimized. Even though you have gobs of CPU and RAM, Apache and MySQL may need tuning. I have seen similar instances to yours, where MySQL hangs due to heavy loads and then incoming HTTP requests stack up. Eventually, connections to the database will block, and slow things down more. When this happens, the system will very often hit swap due to the number of httpd processes. A live webserver hitting swap is a catastrophe; load/memory will swell exponentially.

In this case, you need do a few things.

First, run mysqltuner on your database, and make sure buffer sizes are OK, and that your max memory usage is sane.

Then, tune Apache to make sure you won't ever hit swap under normal conditions by limiting the number of concurrent servers. Enabling keepalives, and setting up some aggressive HTTP cache rules can also help reduce the number of active httpd processes. Using the CDN module can also help reduce Apache load by offloading most CSS, JS, and image requests to the CDN instead of your server.

It is also possible that your memcache doesn't have enough memory, and you are thrashing. You may need to check these settings, and also check which tables you are offloading.

I would also avoid enabling/disabling modules outside of maintenance mode. Schedule windows where you can do work. Assuming you stage/test everything properly, and work from a checklist, you shouldn't have more than 5-10 min downtime to do this.

  • Thanks for the detailed answer. I have run mysqltuner and followed it's suggestions. But I did that before resorting to this forum. In addition ( and I think this is strange ) the server cpu load spikes and does not come down, but memory does not swell and swap remains untouched. So I am stumped as to why that would be the case. Is there a tool for Apache that provides suggestions akin to mysqltuner? I will admit I am a bit of a novice when it comes to configuration. – oranges13 Oct 6 '13 at 0:17
  • APC is handling the Drupal main cache and bootstrap cache; memcache is not directly configured for anything so it should be receiving the other caches ( except the form cache of course ) – oranges13 Oct 6 '13 at 0:21
  • @oranges13 An Apache config tool would be better suited as a question at Server Fault. I just do it manually, and watch top output. I would try using memcache instead of APC for the Drupal cache (but keep APC for opcode caching). APC doesn't have the best cache strategy when it is full; memcache does true LRU. – mpdonadio Oct 6 '13 at 17:50
  • Again, thanks. Your answer makes sense for enabling or disabling a module, but still we should be able to do that without causing the server to spiral into a crash. Additionally, sometimes the same result occurs when we simply view the modules page. Is that still an Apache config problem or something else? – oranges13 Oct 6 '13 at 21:44
  • @oranges13 I had a client with a poorly configured MySQL instance, and Apache configured to allow too many server processes. The server would barf whenever they did a full cache clear. The solution was to tune MySQL and make sure Apache never overloaded the system. My guess is that the cache rebuilds that happen as a result of what I outline are too much for your system to handle. My first step in your case would be to get memcache handling all caches, and drop APC caching of any Drupal data. – mpdonadio Oct 6 '13 at 22:05
1

I wanted to come back and outline what we did to solve this issue.

Firstly, we installed Memcache and APC. Both were provided with as much memory as we could tolerate.

Secondly, we used the Mysql tuning scripts to adjust the MySQL settings. More information about that can be found here: http://www.askapache.com/mysql/performance-tuning-mysql.html

Thirdly, we reconfigured Apache. Internet suggested we add more workers and processes in order to get the server to cooperate. This actually caused more problems because the CPU was struggling to create and destroy the processes. By setting our start servers and Max servers to the same value, it maintains the same amount and does not create churn. By doing this, we were able to actually reduce the number of processes that we were running for Apache, while simultaneously increasing our throughput.

What we learned from this is that there is not a set solution that works. Good luck if you're having these same issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.