0

We have a Drupal Commerce site on D8.6.7 running on Pantheon Performance Small servers. We do a bit of side processing besides normal e-commerce stuff like syncing up data, collecting emails, etc. but nothing immensely crazy. I keep seeing tons of error messages in Pantheon that point to:

  • PhpSerialize
  • EntityStorageBase
  • MemoryCache

From what I can tell, these messages don't even make sense:

Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes)

We believe these are related to errors that are causing interruptions in the Commerce workflow causing certain Drupal events to not fire off. Our Order data is becoming corrupt as a result, e.g. status being reset to Draft even though they've successfully made a payment and the payment amount is being recorded. This results in 'missing' orders that no one sees. I can identify these 'broken' orders via the following SQL query:

select order_id, store_id
from commerce_order 
where commerce_order.state='draft' and commerce_order.total_paid__number is not null;

I cannot determine the origin of this error. I've tried running Webprofiler, I've added logger statements directly into Core, and a whole bunch of digging in various areas. I have yet to reliably reproduce the error and frankly, I'm not a Commerce dev so I don't know how everything is running in there and all the architectural decisions they made over the years.

Given all the possibilities, I don't know if it's even a Commerce issue or if it's related to Pantheon or some combination. Pantheon's error messages are not helpful, I've combed through the Pantheon log messages and lined up timestamps to see if there are any coinciding error messages--none. Same with Watchdog. I've spoken with Pantheon customer support who keeps telling me to search the logs.

I'm at my wits end. I'm basically just stabbing in the dark at this point. If anyone has any tips to debug this, I would very much appreciate it. I've been dealing with this headache for some time now and nobody is happy right now. First occurrence I can remember being from almost back in November. It's actually been highly aggressive recently--what started out as 1-2 incidents per week is now 10-15 per day with no equivalent increase in traffic/orders. Very minor code changes since, just standard updates.

== ADDITIONAL NOTES ==

  • I disabled many non-essential modules last night. Woke up to more error messages still
  • There is a bug that I initially thought was relevant: https://www.drupal.org/project/commerce/issues/3011667 but bojanz told me that it was only affecting OffsitePaymentGateways. We're using Accept.js Authorize.net which extends OnsitePaymentBase
  • Added a logger statement right before return unserialize($raw); in PhpSerialize. We check the strlen to see if it's over 500,000 and then log the strlen & first 10,000 chars. Some logged unserializes are trying to unserialize strings over 3M chars long--I don't want to get too invasive and try to see peak memory usage or anything since that would require me to unserialize before looking at peak memory usage which is starting to change the method call more and more.
| improve this question | |
  • Is there any sort of trace that tells you what function this error occurred in? My gut says if something is trying to read out a 3M (char? MB?) string in a looped process, that could be a good place to start. Are you able to set this up locally and try to replicate it somehow (maybe setup a SUT and create users who are creating orders and try to trigger the memory exhaustion)? – Kevin Feb 3 '19 at 20:47
  • No trace provided unfortunately which I should have mentioned since that makes this that much harder to debug. The error is only happening on Production, we have a CI pipeline that we can test on but I haven't been able to reproduce this bug on other environments. The string that it's trying to unserialize is a 3 million character string, sorry that was ambiguous. I want to see how much memory that is taking up but I don't want to change the PhpSerialize method at all--my loggers just go 'on top' so it doesn't interfere minus using a bit of memory. – cchoe1 Feb 3 '19 at 23:01
  • the serialization.phpserialize service is used to store key_value data (things meant for memory storage like redis/memcache by design if I recall correctly). There might be something in your contrib or custom space using it, but probably not. I only entity-related commerce things in that table. In my local dev env. I found one entry at 33208 bytes long (node.field_storage_definitions). commerce_order's was fairly long too. Also look into if you dev environments are prod-like or have cache/services overridden. – mradcliffe Feb 4 '19 at 1:44
  • I'm not sure how to answer this question because if you're taking a stab in the dark we probably don't have enough information either and will be equally in the dark. – mradcliffe Feb 4 '19 at 1:45
  • I understand that mradcliffe but I do appreciate your response nonetheless. We did have a custom module that was unserializing some data but I don't remember there ever being any custom caching being done. I didn't technically write the module, it was before my time, but I did try to disable it since I saw it was loading in large strings to be unserialized (the 3 million+ length strings). Not sure why it would be caching that anyways but I've disabled it and plan to see what happens. Thanks for the info about phpserialize – cchoe1 Feb 4 '19 at 3:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.