0

I am in the process of developing a new site using Commerce Kickstart.

The site needs to import a fairly large amount of data from a CSV on a regular basis - perhaps 50,000 to 100,000 products at a time, at least every week. (And maybe more often than that.)

I have tried to use Feeds to do a test import of around 30,000 products. (So this creates 30,000 product variations and 30,000 display nodes.) This has proved to be far too slow for my purposes, taking around 18 hours.

So I see my options as follows:

  1. Run some profiling to attempt to see why the import is slowing down so much (It starts off at around 200 products / minute and slows down to 10-20 by the end of the import run.)
  2. Following the strategy here, import into a single table, and use entities to do what I need to do. With this approach, I am concerned about losing or having to re-implement the functionality I need to make the site work - namely faceted search / unknown regular commerce functionality. Each product - each line of the CSV - has numerous fields of data that would have to be stored, so I'm not even sure if this approach would work.

Does anyone have any advice on the best way to progress? Or any hints about the above? Or strategies to import the data that I might not have considered?

Thanks in advance.

EDIT:- Looking at the Feeds log, it looks like the product variations imported in 1 hour 20 mins, whereas the product display nodes took 17 hours to create.

Improve this question
4
  • With modules like Elysia Cron you could set the job of rules to be executed every X minutes, or hour, etc, so it could be faster. Did you try it?
    – arrubiu
    Commented Feb 6, 2014 at 11:50
  • Thanks for the comment. I used Drush to to execute the Feeds import via Elysia Cron.
    – John McCollum
    Commented Feb 6, 2014 at 11:56
  • 1
    The slowness is probably coming from somewhere inside node_save(), I would suggest profiling it and finding out what's going on. The single table strategy won't work with Commerce and makes very little sense there (by bypassing the storage you will also bypass the entire commerce api)
    – Bojan Zivanovic
    Commented Feb 6, 2014 at 14:15
  • Thank you, Bojan. I might be able to use a single table solution - there are around 70 base products that remain the same and can be customised by the 30,000 imported products. (So buy one of 70 items, add a reference to one of the 30,000.) My concern would be that I still need faceted search on there too. At the moment, I'm treating the 30,000 as individual products, with the "base" product as an option.
    – John McCollum
    Commented Feb 6, 2014 at 14:20

1 Answer 1

Reset to default
0

Hopefully this helps someone else.

I used XHProf in an attempt to figure out why the Feeds import was slowing down so much. It appears that the culprit is DrupalDefaultEntityController::cacheGet.

As the number of entries in DrupalDefaultEntityController's entityCache increases, performance drops off a cliff.

I resolved the issue by importing the nodes manually.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.