I am in the process of developing a new site using Commerce Kickstart.
The site needs to import a fairly large amount of data from a CSV on a regular basis - perhaps 50,000 to 100,000 products at a time, at least every week. (And maybe more often than that.)
I have tried to use Feeds to do a test import of around 30,000 products. (So this creates 30,000 product variations and 30,000 display nodes.) This has proved to be far too slow for my purposes, taking around 18 hours.
So I see my options as follows:
- Run some profiling to attempt to see why the import is slowing down so much (It starts off at around 200 products / minute and slows down to 10-20 by the end of the import run.)
- Following the strategy here, import into a single table, and use entities to do what I need to do. With this approach, I am concerned about losing or having to re-implement the functionality I need to make the site work - namely faceted search / unknown regular commerce functionality. Each product - each line of the CSV - has numerous fields of data that would have to be stored, so I'm not even sure if this approach would work.
Does anyone have any advice on the best way to progress? Or any hints about the above? Or strategies to import the data that I might not have considered?
Thanks in advance.
EDIT:- Looking at the Feeds log, it looks like the product variations imported in 1 hour 20 mins, whereas the product display nodes took 17 hours to create.