I try to import a 1.2 GB CSV tab delimited without Headers file as nodes with Drupal Feeds, Feeds Tamper, Feeds Entity Translation.

But it imports only on all my servers 1.799 items (without Erros in Feeds and Drupal logs) and a lot of items are missed. I tried following solutions with Feeds:

  1. Set the max_execution_time to value 3000 in php.ini
  2. Set the php_limit to 256M in php.ini
  3. Set the Feeds process limit from value 50 to 2000 in the settings.php

    $conf['feeds_process_limit'] = 2000;

The Server has 16 GB RAM, SSD etc. - strong Managed Server.

The Import file is saved in the Drupal private Directory on the Server. From this directory i parsing with Feeds.

With small CSV files it works.

Have anyone a better solution or a solution with Feeds?

  • 1
    For large data, command line wins. Consider trying Drush Feeds - don't know if it's good, never used Feeds for large data in any way, but looks promising. – Mołot Oct 23 '14 at 10:45
  • 1
    I just read it on reddit that i should use Drush Feeds and i should set Drush vset http_request_timeout 300 I will try it with Drush. – Alexander Oct 23 '14 at 10:49
  • 2
    Definitely stick with the command line / Drush if possible. – Damien McKenna Oct 23 '14 at 14:26
  • I tried it now with Drush and it imports the same number of items (1799 items). If i use a small 20 MB file with same content only for Germany locations it imports the missed items. – Alexander Oct 25 '14 at 22:44
  • Recently, I faced almost the same issue and I did found out a way how to do this. Is there any manipulation involved in importing the data? – Michiel Jan 23 '15 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.