The site my team is currently building is meant to import a list of all available trips for a travel booking agency.

The method we settled for (for now) can be summarized as such :

  • A daily cron job launches the import and parsing of the XML files containing all availabilities

    • During the parsing of the files, products (Drupal Commerce Product entity) are created, their unique identifier being a 5 letters code
    • Each availability is inserted into a database with its date of departure, its duration, its product code, its price, its original price (if there is a discount), the airport of origin of the flight
    • A queue is then created with each row as an item.
  • The queue (this method was preferred as we were constantly running into memory issues or timeouts) is then launched.

    • it gets data from each column, creates an attribute value (if it does note exist) and then creates a variation for each product. When a new product is loaded into the queue, we only store it’s id and unique reference inside a $product array.

    • Now skipping to the important part : by logging time before and after each operation (loading attribute, creating a variation etc), what has come to my attention is that the process of creating a variation for a given product starts blazingly fast and then gradually slows down to a slug (slowing the whole process to 1.5 seconds). When switching to a new product, the process becomes really fast again.

    Any idea as to why that is ? I am conscious that my product variation table is growing bigger on every item being processed but i’m directly checking the ‘sku’ column of my variations table, not even by product id and it always starts fast when switching to a “fresh” product.

I must add that i noticed that saving the variation also took an increasing amount of time.

(Note : I’m using the entityTypeManager getStorage() method, not Drupal’s database methods).
  • Maybe the same as Memory leak when looping through nodes
    – Hudri
    Jun 21, 2022 at 10:19
  • Thanks for your advice, @Hudri, i implemented the entity storage cache reset in my checkVariation method and it did not seem to increase performance :/
    – BigChing
    Jun 21, 2022 at 12:55
  • 1
    Can't really say why your code is slow without seeing your code.
    – Jaypan
    Jun 21, 2022 at 13:59
  • Did you profile?
    – cilefen
    Jun 21, 2022 at 14:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.