The site my team is currently building is meant to import a list of all available trips for a travel booking agency.
The method we settled for (for now) can be summarized as such :
A daily cron job launches the import and parsing of the XML files containing all availabilities
- During the parsing of the files, products (Drupal Commerce Product entity) are created, their unique identifier being a 5 letters code
- Each availability is inserted into a database with its date of departure, its duration, its product code, its price, its original price (if there is a discount), the airport of origin of the flight
- A queue is then created with each row as an item.
The queue (this method was preferred as we were constantly running into memory issues or timeouts) is then launched.
it gets data from each column, creates an attribute value (if it does note exist) and then creates a variation for each product. When a new product is loaded into the queue, we only store it’s id and unique reference inside a $product array.
Now skipping to the important part : by logging time before and after each operation (loading attribute, creating a variation etc), what has come to my attention is that the process of creating a variation for a given product starts blazingly fast and then gradually slows down to a slug (slowing the whole process to 1.5 seconds). When switching to a new product, the process becomes really fast again.
Any idea as to why that is ? I am conscious that my product variation table is growing bigger on every item being processed but i’m directly checking the ‘sku’ column of my variations table, not even by product id and it always starts fast when switching to a “fresh” product.
I must add that i noticed that saving the variation also took an increasing amount of time.
(Note : I’m using the entityTypeManager getStorage() method, not Drupal’s database methods).
