-2

I've been dealing with this problem for several days. I need to import around 1,600,000 nodes (one time). Everything is set correctly, but the speed is horrible.

Both on the shared server and on the localhost (WAMP and attempts to set the optimal my.ini file). The import is simply horribly slow and after importing 400,000 nodes is even slower.

I understand that processing in Drupal 8 happens per item, but how to set up / edit the import to make it faster?

Well, in the end, I dared to get on the train and started with Migrate CSV and Migrate Tools + Drush.

It looks, it works much better.

7
  • What does slow mean? Set your site in maintenance mode and let it run over night. – leymannx Jan 29 '20 at 7:48
  • 3
    Don't use WAMP, set up a virtual machine with Linux inside. – 4k4 Jan 29 '20 at 7:57
  • Overnight? No, it would spend a week...400,000 nodes have been imported for more than 16 hours. And the next file takes much much longer. – Hefi Jan 29 '20 at 8:21
  • 1
    Are these fairly straightforward nodes, i.e. varchar and number columns? Could you just import them straight into the database, and bypass feeds? – Menno van den Heuvel Jan 29 '20 at 8:27
  • 2
    You're using the Feeds module? Try with Migrate API and Drush. – No Sssweat Jan 29 '20 at 12:47
1

Performance is typically due to the Entity API system operations and the complexity of your migration.

First thing would be to ensure you have nothing enabled like XML Sitemap, Search API / Search indexing, Rules, and other operations that happen on node/entity save unless absolutely necessary (like Pathauto for example).

Second, try leveraging Migrate, Migrate Plus and Migrate CSV instead of Feeds. Grab Migrate Tools so you can run migrations from the command line instead of the UI.

Third, as 4k4 suggested in comments, don't base it only on WAMP / local import. Run this in the environment where the site will be hosted - this will be a much better idea of your performance instead of how it runs on your machine.

You could also try loading the CSV file in and creating a MySQL table out of it, and use the native Sql plugin in Migrate over Migrate CSV. This difference may be negligible, though.

If you are creating/migrating Paragraphs or files (PDFs/images), performance/processing is going to be slower, regardless.

Someone contributed a module to do parallel processing of a migration (ex. 5 at once instead of one at a time) but I can't locate it at the moment. I will update when I do. But modules like this are experimental right now, so YMMV.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.