6

I am using the Batch API in Drupal 7 and I just realized that the reason why my count and totals are not showing correctly is because the initial total is based on a count of the operations given to it.

The thing is I set it up so that each operation processes 100 nodes. I have to update information in nodes that meet certain criteria and there are almost 7,000 nodes that have to be updated. So I thought it would be more efficient to fix multiple nodes per operation call.

Instead it seems like I should put in an operation per node or an operation per 100 nodes (even though my current setup seems to keep calling the same operation with no problems). So what should it be? I guess I could use the context variable to force it to count properly, but I don't like hacking around issues if I don't have to.

Below is the code I am using:

/**
 * Load the nodes with the broken language bodies and fix them.
 */
function my_fixes_broken_bodies() {
  $brokenBodyCnt = my_fixes_count_broken_bodies();
  $operations = array();
  $limit = 100;
  $count = ceil($brokenBodyCnt / $limit);

  // Give helpful information about how many nodes are being operated on.
  drupal_set_message(t('There are @brokenBodyCnt nodes to fix. And we are fixing @limit nodes per call. So the operation will require @count HTTP requests.', array('@brokenBodyCnt' => $brokenBodyCnt, '@limit' => $limit, '@count' => $count)));

  //Setup the operations for the fixes
  $operations[] = array('my_fixes_broken_bodies_op', array('broken body fix'));

  $batch = array(
    'operations' => $operations,
    'finished' => 'my_fixes_finished',
    'title' => t('Fixing broken node body language settings.'),
    'init_message' => t('The broken body fix is starting...'),
    'progress_message' => t('Processed @current out of @total.'),
    'error_message' => t('The broken body fix has encountered an error.'),
  );

  return $batch;
}

/**
 * Batch operation for fixing the broken bodies.
 */
function my_fixes_broken_bodies_op($operation_details, &$context) {
  if (empty($context['sandbox'])) {
    $context['sandbox'] = array();
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_node'] = 0;

    // Save node count for the termination message.
    $context['sandbox']['max'] = my_fixes_count_broken_bodies();
  }

  // Process nodes by groups of 100
  $limit = 100;

  // Retrieve the next group of nids.
  $result = db_query("SELECT n.nid, FROM_UNIXTIME(n.created) AS created, n.title, n.type, n.language AS 'Node Language', b.language AS 'Body Language'
  FROM node n INNER JOIN field_data_body b ON nid = entity_id
  WHERE n.language != b.language AND status = 1
  ORDER BY nid ASC LIMIT $limit");
  $results = $result->fetchAll();

  foreach ($results as $row) {
    $node = node_load($row->nid);

    if (isset($node->body['en']) && !isset($node->body[$node->language])) {
      $node->body[$node->language] = $node->body['en'];
      $node->body['en'] = NULL;
      node_save($node);
    }

    $context['results'][] = $node->nid . ' : ' . check_plain($node->title) . ' ' . $operation_details;

    // Update our progress information.
    $context['sandbox']['progress']++;
    $context['sandbox']['current_node'] = $node->nid;
    $context['message'] = check_plain($node->title);
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = ($context['sandbox']['progress'] >= $context['sandbox']['max']);
  }

  _my_fixes_update_http_requests();
}
3
  • Use $context['sandbox'] to store the total count to process as well as the current nid.
  • Order your SQL by nid ASC
  • Select nodes greater than current nid on next pass.

e.g:

if (!isset($context['sandbox']['max'])) {
  // Set max for progress count later
  $context['sandbox']['max'] = db_query("SELECT count(nid) FROM {node} WHERE type = 'article'")->fetchField();
  $context['sandbox']['current_nid'] = 0;
  $context['sandbox']['progress'] = 0;
}
// Get current set of nodes to process, based on current nid
$limit = 10; // Number to process on each pass
$query = db_select('node', 'n');
$query->fields('n');
$query->condition('type', 'article');
$query->condition('nid', $sandbox['current_nid'], '>');
$query->orderBy('nid');
$query->range(0, $limit);
$result = $query->execute();
foreach($result as $row){
  // Process nodes...
  // Process nodes...
  // Process nodes...      
  $context['message'] = t('Now processing %node', array('%node' => $row->title));
  // Set current node id to start next batch from after this one
  $context['sandbox']['current_nid'] = $row->nid; 
  $context['sandbox']['progress']++;
}
// Inform the batch engine that we are not finished,
// and provide an estimation of the completion level we reached.
if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
  $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
}
  • Unfortunately I am already doing all of this (except the ordering by nid). My problem is the progress reported in the UI is not correct. I will post my code above. – Patrick Jan 23 '13 at 16:17
  • I should add that I have built my module off of the example module's example of using the batch API (batch 2 op). – Patrick Jan 23 '13 at 16:20
3

Ok, so I figured out a better approach. Below I have listed my new code. The gist of it is that I decided to break it out in such a way that I will have one operation per 100 nodes that need to be fixed. I have 6,581 nodes that need to be fixed. So that means I need 66 operations to fix all of the nodes.

I created some calculations that determine how many operations I need to run. Then I set each operation to only process 10 nodes at a time (to save memory). Once an operation has hit it's max of 100 nodes it then tells the Batch API it is finished. This is part of what I was getting hung up on. I thought that max and finished referred to the entire batch process, not the current operation.

Theoretically I could create an operation for every single node that needs to be fixed. But this seems highly excessive, and a bit inefficient. It would increase the db calls quite a bit and the HTTP calls.

Another improvement I made to my code is to only return the node id field in the query instead of all those extra fields that I didn't need. This changed the query run time from about 7.5 seconds to about 0.24 seconds.

All in all I am much happier with the results.

function my_fixes_broken_body_nodes() {
  $results = db_query("SELECT n.nid
  FROM node n INNER JOIN field_data_body b ON nid = entity_id
  WHERE n.language != b.language AND status = 1
  ORDER BY created DESC");

  $count = $results->rowCount();
  $results = $results->fetchAll();

  return array('node_ids' => $results, 'count' => $count);
}

/**
 * Load the nodes with the broken language bodies and fix them.
 */
function my_fixes_broken_bodies() {
  $brokenBodies = my_fixes_broken_body_nodes();
  $brokenBodyCnt = $brokenBodies['count'];
  $operations = array();
  $limit = 10;
  $nodesFixedPerOp = 100;
  $count = ceil($brokenBodyCnt / $limit);
  $opCount = ceil($brokenBodyCnt / $nodesFixedPerOp);

  // Give helpful information about how many nodes are being operated on.
  drupal_set_message(t('There are @brokenBodyCnt nodes to fix. And we are fixing @limit nodes per call. So the operation will require @count HTTP requests.', array('@brokenBodyCnt' => $brokenBodyCnt, '@limit' => $limit, '@count' => $count)));

  //Setup the operations for the fixes
  for ($i = 0; $i < $opCount; $i++) {
    $operations[] = array('my_fixes_broken_bodies_op', array(t('(Operation @operation)', array('@operation' => $i))));
  }

  $batch = array(
    'operations' => $operations,
    'finished' => 'my_fixes_finished',
    'title' => t('Fixing broken node body language settings.'),
    'init_message' => t('The broken body fix is starting...'),
    'progress_message' => t('Processed @current out of @total.'),
    'error_message' => t('The broken body fix has encountered an error.'),
  );

  return $batch;
}

/**
 * Batch operation for fixing the broken bodies.
 */
function my_fixes_broken_bodies_op($operation_details, &$context) {
  // Process nodes by groups of 10
  $limit = 10;

  if (empty($context['sandbox'])) {
    $context['sandbox'] = array();
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_node'] = 0;
    $brokenBodies = my_fixes_broken_body_nodes();

    // Save node count for the termination message.
    $context['sandbox']['max'] = 100;
    $context['sandbox']['node_ids'] = $brokenBodies['node_ids'];
  }

  // Retrieve the next group of nids.
  for ($i = 0; $i < $limit; $i++) {
    $row = array_shift($context['sandbox']['node_ids']);
    $node = NULL;

    //Load the node
    if (isset($row->nid) && !empty($row->nid)) {
      $node = node_load($row->nid);
    }

    //Make sure it loaded alright and then run operations on it.
    if (!empty($node) && is_object($node)) {
      if (isset($node->body['en']) && !isset($node->body[$node->language])) {
        $node->body[$node->language] = $node->body['en'];
        $node->body['en'] = NULL;
        node_save($node);
      }

      $context['results'][] = $node->nid . ' : ' . check_plain($node->title) . ' ' . $operation_details;

      // Update our progress information.
      $context['sandbox']['progress']++;
      $context['sandbox']['current_node'] = $node->nid;
      $context['message'] = check_plain($node->title);
    }
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = ($context['sandbox']['progress'] >= $context['sandbox']['max']);
  }

  _my_fixes_update_http_requests();
}
  • Your code is very useful. However, in my situation I need to process up to 50 thousand to 100 thousand nodes. Your solution loads initially all the rows out of the SQL database. I wonder if it is possible to load the data from the database inside the batch operation utilizing MySQL's LIMIT and OFFSET operators. Is there any documentation available regarding this approach? – W.M. Feb 20 '17 at 13:38
  • 1
    You would still need to get a count of how many there are to setup the batch properly. But after that you could load via DB operations. – Patrick May 18 '17 at 17:42
  • The for loop inside of my_fixes_broken_bodies() adding a separate instance of my_fixes_broken_bodies_op() per chunk is unnecessary. The point of the batch API is iterating over the same operation multiple times. $context['finished'] stores the progress (between 0 and 1) through the batch and as long as it is set and less than 1, then the operation will repeat. Thus, all the logic for querying your nodes can be in the my_fixes_broken_bodies_op() function and limited to the current set of nodes. Get the count as a separate query inside the if (empty($context['sandbox'])) {... block. – Sean Adams-Hiett Jul 27 '18 at 14:02
0

I'm looking for same answer and I think array_chunk() is a good idea for meet this requirement. See this example:http://www.metaltoad.com/blog/using-drupal-batch-api

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.