We have a Drupal Commerce site on D8.6.7 running on Pantheon Performance Small servers. We do a bit of side processing besides normal e-commerce stuff like syncing up data, collecting emails, etc. but nothing immensely crazy. I keep seeing tons of error messages in Pantheon that point to:
From what I can tell, these messages don't even make sense:
Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes)
We believe these are related to errors that are causing interruptions in the Commerce workflow causing certain Drupal events to not fire off. Our Order data is becoming corrupt as a result, e.g. status being reset to Draft even though they've successfully made a payment and the payment amount is being recorded. This results in 'missing' orders that no one sees. I can identify these 'broken' orders via the following SQL query:
select order_id, store_id from commerce_order where commerce_order.state='draft' and commerce_order.total_paid__number is not null;
I cannot determine the origin of this error. I've tried running Webprofiler, I've added logger statements directly into Core, and a whole bunch of digging in various areas. I have yet to reliably reproduce the error and frankly, I'm not a Commerce dev so I don't know how everything is running in there and all the architectural decisions they made over the years.
Given all the possibilities, I don't know if it's even a Commerce issue or if it's related to Pantheon or some combination. Pantheon's error messages are not helpful, I've combed through the Pantheon log messages and lined up timestamps to see if there are any coinciding error messages--none. Same with Watchdog. I've spoken with Pantheon customer support who keeps telling me to search the logs.
I'm at my wits end. I'm basically just stabbing in the dark at this point. If anyone has any tips to debug this, I would very much appreciate it. I've been dealing with this headache for some time now and nobody is happy right now. First occurrence I can remember being from almost back in November. It's actually been highly aggressive recently--what started out as 1-2 incidents per week is now 10-15 per day with no equivalent increase in traffic/orders. Very minor code changes since, just standard updates.
== ADDITIONAL NOTES ==
- I disabled many non-essential modules last night. Woke up to more error messages still
- There is a bug that I initially thought was relevant: https://www.drupal.org/project/commerce/issues/3011667 but bojanz told me that it was only affecting OffsitePaymentGateways. We're using Accept.js Authorize.net which extends OnsitePaymentBase
- Added a logger statement right before
return unserialize($raw);in PhpSerialize. We check the strlen to see if it's over 500,000 and then log the strlen & first 10,000 chars. Some logged unserializes are trying to unserialize strings over 3M chars long--I don't want to get too invasive and try to see peak memory usage or anything since that would require me to unserialize before looking at peak memory usage which is starting to change the method call more and more.