When using the Batch API should I create an operation per node?

Question

I am using the Batch API in Drupal 7 and I just realized that the reason why my count and totals are not showing correctly is because the initial total is based on a count of the operations given to it.

The thing is I set it up so that each operation processes 100 nodes. I have to update information in nodes that meet certain criteria and there are almost 7,000 nodes that have to be updated. So I thought it would be more efficient to fix multiple nodes per operation call.

Instead it seems like I should put in an operation per node or an operation per 100 nodes (even though my current setup seems to keep calling the same operation with no problems). So what should it be? I guess I could use the context variable to force it to count properly, but I don't like hacking around issues if I don't have to.

Below is the code I am using:

/**
 * Load the nodes with the broken language bodies and fix them.
 */
function my_fixes_broken_bodies() {
  $brokenBodyCnt = my_fixes_count_broken_bodies();
  $operations = array();
  $limit = 100;
  $count = ceil($brokenBodyCnt / $limit);

  // Give helpful information about how many nodes are being operated on.
  drupal_set_message(t('There are @brokenBodyCnt nodes to fix. And we are fixing @limit nodes per call. So the operation will require @count HTTP requests.', array('@brokenBodyCnt' => $brokenBodyCnt, '@limit' => $limit, '@count' => $count)));

  //Setup the operations for the fixes
  $operations[] = array('my_fixes_broken_bodies_op', array('broken body fix'));

  $batch = array(
    'operations' => $operations,
    'finished' => 'my_fixes_finished',
    'title' => t('Fixing broken node body language settings.'),
    'init_message' => t('The broken body fix is starting...'),
    'progress_message' => t('Processed @current out of @total.'),
    'error_message' => t('The broken body fix has encountered an error.'),
  );

  return $batch;
}

/**
 * Batch operation for fixing the broken bodies.
 */
function my_fixes_broken_bodies_op($operation_details, &$context) {
  if (empty($context['sandbox'])) {
    $context['sandbox'] = array();
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_node'] = 0;

    // Save node count for the termination message.
    $context['sandbox']['max'] = my_fixes_count_broken_bodies();
  }

  // Process nodes by groups of 100
  $limit = 100;

  // Retrieve the next group of nids.
  $result = db_query("SELECT n.nid, FROM_UNIXTIME(n.created) AS created, n.title, n.type, n.language AS 'Node Language', b.language AS 'Body Language'
  FROM node n INNER JOIN field_data_body b ON nid = entity_id
  WHERE n.language != b.language AND status = 1
  ORDER BY nid ASC LIMIT $limit");
  $results = $result->fetchAll();

  foreach ($results as $row) {
    $node = node_load($row->nid);

    if (isset($node->body['en']) && !isset($node->body[$node->language])) {
      $node->body[$node->language] = $node->body['en'];
      $node->body['en'] = NULL;
      node_save($node);
    }

    $context['results'][] = $node->nid . ' : ' . check_plain($node->title) . ' ' . $operation_details;

    // Update our progress information.
    $context['sandbox']['progress']++;
    $context['sandbox']['current_node'] = $node->nid;
    $context['message'] = check_plain($node->title);
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = ($context['sandbox']['progress'] >= $context['sandbox']['max']);
  }

  _my_fixes_update_http_requests();
}

Answer 1

Ok, so I figured out a better approach. Below I have listed my new code. The gist of it is that I decided to break it out in such a way that I will have one operation per 100 nodes that need to be fixed. I have 6,581 nodes that need to be fixed. So that means I need 66 operations to fix all of the nodes.

I created some calculations that determine how many operations I need to run. Then I set each operation to only process 10 nodes at a time (to save memory). Once an operation has hit it's max of 100 nodes it then tells the Batch API it is finished. This is part of what I was getting hung up on. I thought that max and finished referred to the entire batch process, not the current operation.

Theoretically I could create an operation for every single node that needs to be fixed. But this seems highly excessive, and a bit inefficient. It would increase the db calls quite a bit and the HTTP calls.

Another improvement I made to my code is to only return the node id field in the query instead of all those extra fields that I didn't need. This changed the query run time from about 7.5 seconds to about 0.24 seconds.

All in all I am much happier with the results.

function my_fixes_broken_body_nodes() {
  $results = db_query("SELECT n.nid
  FROM node n INNER JOIN field_data_body b ON nid = entity_id
  WHERE n.language != b.language AND status = 1
  ORDER BY created DESC");

  $count = $results->rowCount();
  $results = $results->fetchAll();

  return array('node_ids' => $results, 'count' => $count);
}

/**
 * Load the nodes with the broken language bodies and fix them.
 */
function my_fixes_broken_bodies() {
  $brokenBodies = my_fixes_broken_body_nodes();
  $brokenBodyCnt = $brokenBodies['count'];
  $operations = array();
  $limit = 10;
  $nodesFixedPerOp = 100;
  $count = ceil($brokenBodyCnt / $limit);
  $opCount = ceil($brokenBodyCnt / $nodesFixedPerOp);

  // Give helpful information about how many nodes are being operated on.
  drupal_set_message(t('There are @brokenBodyCnt nodes to fix. And we are fixing @limit nodes per call. So the operation will require @count HTTP requests.', array('@brokenBodyCnt' => $brokenBodyCnt, '@limit' => $limit, '@count' => $count)));

  //Setup the operations for the fixes
  for ($i = 0; $i < $opCount; $i++) {
    $operations[] = array('my_fixes_broken_bodies_op', array(t('(Operation @operation)', array('@operation' => $i))));
  }

  $batch = array(
    'operations' => $operations,
    'finished' => 'my_fixes_finished',
    'title' => t('Fixing broken node body language settings.'),
    'init_message' => t('The broken body fix is starting...'),
    'progress_message' => t('Processed @current out of @total.'),
    'error_message' => t('The broken body fix has encountered an error.'),
  );

  return $batch;
}

/**
 * Batch operation for fixing the broken bodies.
 */
function my_fixes_broken_bodies_op($operation_details, &$context) {
  // Process nodes by groups of 10
  $limit = 10;

  if (empty($context['sandbox'])) {
    $context['sandbox'] = array();
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_node'] = 0;
    $brokenBodies = my_fixes_broken_body_nodes();

    // Save node count for the termination message.
    $context['sandbox']['max'] = 100;
    $context['sandbox']['node_ids'] = $brokenBodies['node_ids'];
  }

  // Retrieve the next group of nids.
  for ($i = 0; $i < $limit; $i++) {
    $row = array_shift($context['sandbox']['node_ids']);
    $node = NULL;

    //Load the node
    if (isset($row->nid) && !empty($row->nid)) {
      $node = node_load($row->nid);
    }

    //Make sure it loaded alright and then run operations on it.
    if (!empty($node) && is_object($node)) {
      if (isset($node->body['en']) && !isset($node->body[$node->language])) {
        $node->body[$node->language] = $node->body['en'];
        $node->body['en'] = NULL;
        node_save($node);
      }

      $context['results'][] = $node->nid . ' : ' . check_plain($node->title) . ' ' . $operation_details;

      // Update our progress information.
      $context['sandbox']['progress']++;
      $context['sandbox']['current_node'] = $node->nid;
      $context['message'] = check_plain($node->title);
    }
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = ($context['sandbox']['progress'] >= $context['sandbox']['max']);
  }

  _my_fixes_update_http_requests();
}

Answer 2

• Use $context['sandbox'] to store the total count to process as well as the current nid.
• Order your SQL by nid ASC
• Select nodes greater than current nid on next pass.

e.g:

if (!isset($context['sandbox']['max'])) {
  // Set max for progress count later
  $context['sandbox']['max'] = db_query("SELECT count(nid) FROM {node} WHERE type = 'article'")->fetchField();
  $context['sandbox']['current_nid'] = 0;
  $context['sandbox']['progress'] = 0;
}
// Get current set of nodes to process, based on current nid
$limit = 10; // Number to process on each pass
$query = db_select('node', 'n');
$query->fields('n');
$query->condition('type', 'article');
$query->condition('nid', $sandbox['current_nid'], '>');
$query->orderBy('nid');
$query->range(0, $limit);
$result = $query->execute();
foreach($result as $row){
  // Process nodes...
  // Process nodes...
  // Process nodes...      
  $context['message'] = t('Now processing %node', array('%node' => $row->title));
  // Set current node id to start next batch from after this one
  $context['sandbox']['current_nid'] = $row->nid; 
  $context['sandbox']['progress']++;
}
// Inform the batch engine that we are not finished,
// and provide an estimation of the completion level we reached.
if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
  $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
}

Answer 3

I'm looking for same answer and I think array_chunk() is a good idea for meet this requirement. See this example:http://www.metaltoad.com/blog/using-drupal-batch-api