1

I try to import a 1.2 GB CSV tab delimited without Headers file as nodes with Drupal Feeds, Feeds Tamper, Feeds Entity Translation.

But it imports only on all my servers 1.799 items (without Erros in Feeds and Drupal logs) and a lot of items are missed. I tried following solutions with Feeds:

  1. Set the max_execution_time to value 3000 in php.ini
  2. Set the php_limit to 256M in php.ini
  3. Set the Feeds process limit from value 50 to 2000 in the settings.php

    $conf['feeds_process_limit'] = 2000;

The Server has 16 GB RAM, SSD etc. - strong Managed Server.

The Import file is saved in the Drupal private Directory on the Server. From this directory i parsing with Feeds.

With small CSV files it works.

Have anyone a better solution or a solution with Feeds?

  • 1
    For large data, command line wins. Consider trying Drush Feeds - don't know if it's good, never used Feeds for large data in any way, but looks promising. – Mołot Oct 23 '14 at 10:45
  • 1
    I just read it on reddit that i should use Drush Feeds and i should set Drush vset http_request_timeout 300 I will try it with Drush. – Alexander Oct 23 '14 at 10:49
  • 2
    Definitely stick with the command line / Drush if possible. – Damien McKenna Oct 23 '14 at 14:26
  • I tried it now with Drush and it imports the same number of items (1799 items). If i use a small 20 MB file with same content only for Germany locations it imports the missed items. – Alexander Oct 25 '14 at 22:44
  • 1
    I solved it now with a custom Import PHP Script. My strategy was to split the big CSV file with Python into a lot of small files. 4.000 lines per file and importing them via Drush. – Alexander Jan 24 '15 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.