I try to import a 1.2 GB CSV tab delimited without Headers file as nodes with Drupal Feeds, Feeds Tamper, Feeds Entity Translation.

But it imports only on all my servers 1.799 items (without Erros in Feeds and Drupal logs) and a lot of items are missed. I tried following solutions with Feeds:

  1. Set the max_execution_time to value 3000 in php.ini
  2. Set the php_limit to 256M in php.ini
  3. Set the Feeds process limit from value 50 to 2000 in the settings.php

    $conf['feeds_process_limit'] = 2000;

The Server has 16 GB RAM, SSD etc. - strong Managed Server.

The Import file is saved in the Drupal private Directory on the Server. From this directory i parsing with Feeds.

With small CSV files it works.

Have anyone a better solution or a solution with Feeds?

  • 1
    For large data, command line wins. Consider trying Drush Feeds - don't know if it's good, never used Feeds for large data in any way, but looks promising.
    – Mołot
    Oct 23, 2014 at 10:45
  • 1
    I just read it on reddit that i should use Drush Feeds and i should set Drush vset http_request_timeout 300 I will try it with Drush.
    – Alexander
    Oct 23, 2014 at 10:49
  • 2
    Definitely stick with the command line / Drush if possible. Oct 23, 2014 at 14:26
  • I tried it now with Drush and it imports the same number of items (1799 items). If i use a small 20 MB file with same content only for Germany locations it imports the missed items.
    – Alexander
    Oct 25, 2014 at 22:44
  • 1
    I solved it now with a custom Import PHP Script. My strategy was to split the big CSV file with Python into a lot of small files. 4.000 lines per file and importing them via Drush.
    – Alexander
    Jan 24, 2015 at 20:49


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.