I was asked to help with a Drupal 7 site that I've helped to tweak several times before. The problem is, most of the site pages are not shown and "Internal Server Error" 500 page is shown instead (it started at the 6th of may). There are some exceptions though:
/user (but not /user/1 for example) /admin /admin/content /admin/structure /admin/appearance /admin/people /admin/modules (but not /admin/config or /admin/reports)
(may be these are not all the exceptions, but I don't know others yet)
Now, here's what I know (I don't have full root access, but I can request access for some resources):
PHPMyAdmin shows over 10K requests per minute (84% are
select, 6% delete, 6% insert, ..). By the way, it also shows that it was started at the 7th of May (may be administrator has rebooted it)
httpd access_log consists of lines like (I splitted into 2 for readability)
::1 - - [13/May/2020:02:07:32 +0300] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/5.4.16 (internal dummy connection)"
there's nothing except these ones (but I did some attempts to open the site during that period); these ones are added with frequency of about 5-20 lines per second
httpd error_log consists mostly of lines like
[Wed May 13 09:16:09.487641 2020] [fcgid:warn] [pid 27860] mod_fcgid: process 25947 graceful kill fail, sending SIGKILL
and these are added even less frequently: about 10 times per hour
processor is highly loaded: it varies from 10% to 75% CPU consumed by
/usr/bin/php-cgi php(which can be seen by using
/usr/bin/php-cgi php; usage is increased when I try to load a page, I've seen it bumping from 10% to 50% in my manual tests)
Looks like something's wrong with mod_fcgid, but I have very little idea of what I can do to debug and fix this. Can anybody help with this?