1

Ok, so I've got a script that's going to go through about 13000 nodes (which might be part of the problem, but I don't think it is).

I can either put them as one large array, which seems to go through the nodes fairly quickly, but eventually leads to a timeout after 600 seconds (10 minutes).

My batch is being produced with this code:

function revision_published_lr_build_batch($items) {

   $count_items = count($items);

    $i = 0;

   foreach ($items as $item) {
    $i++;
    // Here we can add multiple operation using an array variable.
    $operations[] = array('revision_published_process_data', array($item, 'details' => t('(Importing item @item  of  @count)', array('@item ' => $i, '@count' => $count_items)))); // operation with argument
    } 
  //$operations[] = array('revision_published_process_data', array($items, 'details' => t('(Importing items)')));
  //Define your batch operation here
  $batch = array(
    'title' => t('Batch operation process'),
    'operations' => $operations,
    'finished' => 'revision_published_lr_build_batch_finished',
    'init_message' => t('Initializing...'),
    'progress_message' => t('Operation @current out of @total.'),
    'error_message' => t('Found some error here.'),
  );

  return $batch;
}

Each batch item should be just an MD5 hash.

However, when going through everything, it is taking forever - the process has been running for hours and it is only on the 86th or so node.

Note, the ONLY major difference in speed appears to be whether or not I put in the foreach loop and create a lot of different operations, or whether I just put $items straight into the operations variable.

Why would this difference cause such a marked speed difference? Batch API seems like it would be more suited to work with lots of small items, rather than one big one.

I suppose one solution would be putting 100 nodes in one batch item, increasing speed for processing and decreasing the probability of a timeout during processing, but this seems like an inelegant solution.

So in short - how do I make batch API process faster?

To clarify, changing the code to this (and removing the foreach loop) is what makes processing faster (but eventually leads to a timeout because the processing is all happening at once):

$operations[] = array('revision_published_process_data', array($items, 'details' => t('(Importing item @item  of  @count)', array('@item ' => $i, '@count' => $count_items))));
3

You're asking alot of questions ... here are some general answers.

  • The batch api is used to take a set of data and partition out the data into workable batches. The batch api does not guarentee that timeouts will not occur. you need to find a batch size that that is the sweet spot for your dataset size.
  • typically you would choose a batch size of say 5, 10, 15 or 20 items. and called node_save for example 25 times before going onto the next batch (page refresh). The page refreshes per batch do slow down your import process ... due to the HTTP requests.
  • if you're querying data say from a Queue or something if you can try to avoid re-fetching the data and using a static php variable to store information you can save some time.
  • The Migrate module is designed to solve these kind of problems by using Drush.

If ordering of the data doesnt matter. I find parallelizing (eg, multiple worker threads) can help with these types of imports look at Background Process and Ultimate Cron Queue Scalar.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.