How to speed up Batch API processing

Question

Ok, so I've got a script that's going to go through about 13000 nodes (which might be part of the problem, but I don't think it is).

I can either put them as one large array, which seems to go through the nodes fairly quickly, but eventually leads to a timeout after 600 seconds (10 minutes).

My batch is being produced with this code:

function revision_published_lr_build_batch($items) {

   $count_items = count($items);

    $i = 0;

   foreach ($items as $item) {
    $i++;
    // Here we can add multiple operation using an array variable.
    $operations[] = array('revision_published_process_data', array($item, 'details' => t('(Importing item @item  of  @count)', array('@item ' => $i, '@count' => $count_items)))); // operation with argument
    } 
  //$operations[] = array('revision_published_process_data', array($items, 'details' => t('(Importing items)')));
  //Define your batch operation here
  $batch = array(
    'title' => t('Batch operation process'),
    'operations' => $operations,
    'finished' => 'revision_published_lr_build_batch_finished',
    'init_message' => t('Initializing...'),
    'progress_message' => t('Operation @current out of @total.'),
    'error_message' => t('Found some error here.'),
  );

  return $batch;
}

Each batch item should be just an MD5 hash.

However, when going through everything, it is taking forever - the process has been running for hours and it is only on the 86th or so node.

Note, the ONLY major difference in speed appears to be whether or not I put in the foreach loop and create a lot of different operations, or whether I just put $items straight into the operations variable.

Why would this difference cause such a marked speed difference? Batch API seems like it would be more suited to work with lots of small items, rather than one big one.

I suppose one solution would be putting 100 nodes in one batch item, increasing speed for processing and decreasing the probability of a timeout during processing, but this seems like an inelegant solution.

So in short - how do I make batch API process faster?

To clarify, changing the code to this (and removing the foreach loop) is what makes processing faster (but eventually leads to a timeout because the processing is all happening at once):

$operations[] = array('revision_published_process_data', array($items, 'details' => t('(Importing item @item  of  @count)', array('@item ' => $i, '@count' => $count_items))));

Answer 1

You're asking alot of questions ... here are some general answers.

• The batch api is used to take a set of data and partition out the data into workable batches. The batch api does not guarentee that timeouts will not occur. you need to find a batch size that that is the sweet spot for your dataset size.
• typically you would choose a batch size of say 5, 10, 15 or 20 items. and called node_save for example 25 times before going onto the next batch (page refresh). The page refreshes per batch do slow down your import process ... due to the HTTP requests.
• if you're querying data say from a Queue or something if you can try to avoid re-fetching the data and using a static php variable to store information you can save some time.
• The Migrate module is designed to solve these kind of problems by using Drush.

If ordering of the data doesnt matter. I find parallelizing (eg, multiple worker threads) can help with these types of imports look at Background Process and Ultimate Cron Queue Scalar.