I setup my Drupal 7 feed importer to import from a csv file with 20,000 nodes. I'm able to get it working with 20 nodes just to test to make sure everything is working. But is there a setting to be able to process X amount of nodes at once? My csv file is 12MB and I'd like to just do 300 at a time, but I don't see an option for this anywhere.
How would I make it process a set amount?
Recent versions of Feeds (since 7.x-2.0-beta1) will try to import as many items as it can per cron run. Imports can be ran on cron runs by either using the "Process in background" option or by setting the "Periodic import" option. By default, Feeds indeed imports items in batches of 50, but on each cron run Feeds will try to run as many batches as it can one minute.
If you want Feeds to run longer than one minute on each cron run, you can do so by altering the time limit of the queue 'feeds_source_import' by implementing hook_cron_queue_info_alter():
/**
* Implements hook_cron_queue_info_alter().
*/
function mymodule_cron_queue_info_alter(&$queues) {
// Run Feeds imports at max. 5 minutes per cron run.
$queues['feeds_source_import']['time'] = 300;
}
You can also increase the batch size by setting the 'feeds_process_limit' variable:
// Process 250 items per batch.
variable_set('feeds_process_limit', 250);
Do so only if you are using an alternative way of importing the data (as on cron runs Feeds will already try to import multiple batches).
Finally, the dev version of Feeds has improved support for larger files downloaded with the HTTP Fetcher. I'm still working on the issue that a large CSV file must be completely imported within 6 hours. If you need more than 6 hours to complete the import (for example: if cron doesn't run often), try the patch from https://www.drupal.org/node/1029102.
drush vset, or by setting it in your settings.php file: $conf['feeds_process_limit'] = 100;. Changing it in plugins/FeedsProcessor.inc is highly discouraged, for reasons see drupal.org/docs/7/site-building-best-practices/never-hack-core.
Commented
Aug 31, 2017 at 9:43
Which version of Drupal are you using? I haven't checked on Feeds for Drupal 8, but Feeds for Drupal 7 (unless it's changed in the last few years) has a hard-coded item limit of 50. If you're importing from a CSV, your node architecture is probably pretty simple. You could try the Feed Import module instead of Feeds, in my experience it's better suited for very large imports.
