How to import a million of nodes? Very slow import

Question

I've been dealing with this problem for several days. I need to import around 1,600,000 nodes (one time). Everything is set correctly, but the speed is horrible.

Both on the shared server and on the localhost (WAMP and attempts to set the optimal my.ini file). The import is simply horribly slow and after importing 400,000 nodes is even slower.

I understand that processing in Drupal 8 happens per item, but how to set up / edit the import to make it faster?

Well, in the end, I dared to get on the train and started with Migrate CSV and Migrate Tools + Drush.

It looks, it works much better.

Answer 1

Performance is typically due to the Entity API system operations and the complexity of your migration.

First thing would be to ensure you have nothing enabled like XML Sitemap, Search API / Search indexing, Rules, and other operations that happen on node/entity save unless absolutely necessary (like Pathauto for example).

Second, try leveraging Migrate, Migrate Plus and Migrate CSV instead of Feeds. Grab Migrate Tools so you can run migrations from the command line instead of the UI.

Third, as 4k4 suggested in comments, don't base it only on WAMP / local import. Run this in the environment where the site will be hosted - this will be a much better idea of your performance instead of how it runs on your machine.

You could also try loading the CSV file in and creating a MySQL table out of it, and use the native Sql plugin in Migrate over Migrate CSV. This difference may be negligible, though.

If you are creating/migrating Paragraphs or files (PDFs/images), performance/processing is going to be slower, regardless.

Someone contributed a module to do parallel processing of a migration (ex. 5 at once instead of one at a time) but I can't locate it at the moment. I will update when I do. But modules like this are experimental right now, so YMMV.