CPU spikes enabling / disabling modules when site is under load

Question

We are in the process of migrating our website to Drupal 7 and we had a failed launch this past week. We performed load testing and were certain that our server was going to work just fine with our current load, unfortunately we did not perform load testing with logged-in users (we use a 3rd party gateway login system and it was almost impossible to accomplish a load test this way). Long story short - logged in users overloaded our site and the server crashed.

So we've gone back to the drawing board, implemented APC and Memcache and we believe we have the problem fixed. The CPU no longer maxes out when under an expected load.

However, we now have a new issue that is a critical showstopper if we cannot find a fix for it.

While the server is under load from users, if an administrator attempts to enable or disable a module, ANY MODULE, (and sometimes even if we attempt to VIEW the modules page), the CPU will max out and the server will then eventually crash. The only option at that point is to restart the httpd service, which of course disrupts all user traffic.

When operating normally, CPU usage is at about 20 - 30%. When you attempt to enable or disable a module, the server IMMEDIATELY pegs the CPU at 100% and becomes nearly unresponsive from the browser and command line. Our normal page load times are 400ms, and they skyrocket to over a minute until the server crashes.

With no user load, the server operates as expected, though the action of enabling / disabling a module does spike the CPU for a short period of time. I have checked the sql slow query log and haven't found anything enlightening there. HTTPD is not spitting out any errors either.

Server has 2 CPU, 16GB memory and is dedicated to running this site alone; surely it isn't under powered.

What could possibly be causing this problem?

Answer 1

If you look at the source for module_enable() you can see what is going on. In particular, several cache tables get cleared, as does the registry. It also does filesystem scanning to rebuild module data. All of these are expensive operations.

If you look at the source for system_modules() (this is the callback for the modules page), you will see that system_rebuild_module_data() is called. This is also an expensive operation.

My guess is that something on your server isn't optimized. Even though you have gobs of CPU and RAM, Apache and MySQL may need tuning. I have seen similar instances to yours, where MySQL hangs due to heavy loads and then incoming HTTP requests stack up. Eventually, connections to the database will block, and slow things down more. When this happens, the system will very often hit swap due to the number of httpd processes. A live webserver hitting swap is a catastrophe; load/memory will swell exponentially.

In this case, you need do a few things.

First, run mysqltuner on your database, and make sure buffer sizes are OK, and that your max memory usage is sane.

Then, tune Apache to make sure you won't ever hit swap under normal conditions by limiting the number of concurrent servers. Enabling keepalives, and setting up some aggressive HTTP cache rules can also help reduce the number of active httpd processes. Using the CDN module can also help reduce Apache load by offloading most CSS, JS, and image requests to the CDN instead of your server.

It is also possible that your memcache doesn't have enough memory, and you are thrashing. You may need to check these settings, and also check which tables you are offloading.

I would also avoid enabling/disabling modules outside of maintenance mode. Schedule windows where you can do work. Assuming you stage/test everything properly, and work from a checklist, you shouldn't have more than 5-10 min downtime to do this.

Answer 2

I wanted to come back and outline what we did to solve this issue.

Firstly, we installed Memcache and APC. Both were provided with as much memory as we could tolerate.

Secondly, we used the Mysql tuning scripts to adjust the MySQL settings. More information about that can be found here: http://www.askapache.com/mysql/performance-tuning-mysql.html

Thirdly, we reconfigured Apache. Internet suggested we add more workers and processes in order to get the server to cooperate. This actually caused more problems because the CPU was struggling to create and destroy the processes. By setting our start servers and Max servers to the same value, it maintains the same amount and does not create churn. By doing this, we were able to actually reduce the number of processes that we were running for Apache, while simultaneously increasing our throughput.

What we learned from this is that there is not a set solution that works. Good luck if you're having these same issues.