We are in the process of migrating our website to Drupal 7 and we had a failed launch this past week. We performed load testing and were certain that our server was going to work just fine with our current load, unfortunately we did not perform load testing with logged-in users (we use a 3rd party gateway login system and it was almost impossible to accomplish a load test this way). Long story short - logged in users overloaded our site and the server crashed.
So we've gone back to the drawing board, implemented APC and Memcache and we believe we have the problem fixed. The CPU no longer maxes out when under an expected load.
However, we now have a new issue that is a critical showstopper if we cannot find a fix for it.
While the server is under load from users, if an administrator attempts to enable or disable a module, ANY MODULE, (and sometimes even if we attempt to VIEW the modules page), the CPU will max out and the server will then eventually crash. The only option at that point is to restart the httpd service, which of course disrupts all user traffic.
When operating normally, CPU usage is at about 20 - 30%. When you attempt to enable or disable a module, the server IMMEDIATELY pegs the CPU at 100% and becomes nearly unresponsive from the browser and command line. Our normal page load times are 400ms, and they skyrocket to over a minute until the server crashes.
With no user load, the server operates as expected, though the action of enabling / disabling a module does spike the CPU for a short period of time. I have checked the sql slow query log and haven't found anything enlightening there. HTTPD is not spitting out any errors either.
Server has 2 CPU, 16GB memory and is dedicated to running this site alone; surely it isn't under powered.
What could possibly be causing this problem?