How do I structure the batch API array?

Question

I'm working on a Drupal 8 custom module to scrape pages from a site for migration. Since we're dealing with a large number of pages, I wanted to set up batch processing, to deal with timeouts.

I looked at two tutorials, this one, and one here. However, when I ran my batch, it didn't work as expected. The progress bar immediately filled up to all 71 items, and there were no imported pages, but a number of errors in the error log.

To rule out confounding factors, I created a simple module for testing the batch api (I didn't find the batch represented in the D8 examples module). It exhibits the same behavior: it immediately processes all items, and there are no successes, just errors. This an the example error:

 Warning: call_user_func_array() expects parameter 1 to be a valid callback, class '\Drupal\batch_test\BatchTestForm' not found in _batch_process() (line 252 of C:\wamp64\www\D8\core\includes\batch.inc)  
`#0` C:\wamp64\www\D8\core\includes\bootstrap.inc(566): _drupal_error_handler_real(2, 'call_user_func_...', 'C:\\wamp64\\www\\p...', 252, Array)  
`#1` [internal function]: _drupal_error_handler(2, 'call_user_func_...', 'C:\\wamp64\\www\\p...', 252, Array)  
`#2` C:\wamp64\www\D8\core\includes\batch.inc(252): call_user_func_array('\\Drupal\\batch_t...', NULL)  
`#3` C:\wamp64\www\D8\core\includes\batch.inc(95): _batch_process()  
`#4` C:\wamp64\www\D8\core\includes\batch.inc(77): _batch_do()  
`#5` C:\wamp64\www\D8\core\modules\system\src\Controller\BatchController.php(55): _batch_page(Object(Symfony\Component\HttpFoundation\Request))  
`#6` [internal function]: Drupal\system\Controller\BatchController->batchPage(Object(Symfony\Component\HttpFoundation\Request))  
`#7` C:\wamp64\www\D8\core\lib\Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array(Array, Array)  
`#8` C:\wamp64\www\D8\core\lib\Drupal\Core\Render\Renderer.php(574): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()

The following is the form that I'm using. Both of the tutorials indicate that calling _batch_process() is not necessary when calling batch_set() in a form submit.

namespace Drupal\batch_test\Form;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Form\FormBase;

class BatchTestForm extends FormBase {

  public function getFormId() {
    return 'batch_test_admin_form';
  }

  public function buildForm(array $form, FormStateInterface $form_state) {
    $form['submit'] = [
      '#type' => 'submit',
      '#value' => $this->t('Test Batch'),
      '#button_type' => 'primary',
    ];
    return $form;
  }

  public function submitForm(array &$form, FormStateInterface $form_state) {
    $batch = array(
      'title' => t('Test Batch'),
      'operations' => [],
      'finished' => 'Drupal\BatchTest\Form\batchCallback',
      'progress_message' => t('Processed @current out of @total.'),
    );

    for ( $i = 0; $i <= mt_rand(10,100); $i++ ) {
      $batch['operations'][] = ['\Drupal\batch_test\BatchTestForm::process',$i];
    }

    batch_set($batch);
  }

  public function process($input, &$context) {
    // log a random string
    \Drupal::logger('batch_test')->notice(base64_encode(random_bytes(10)));
    sleep(mt_rand(1,100));
  }

  public function batchCallback($success, $results, $operations) {
    if ($success) {
      $message = \Drupal::translation()->formatPlural(
        count($results),
        'One page scraped.', '@count pages scraped.'
      );
    }
    else {
      $message = t('Finished with an error.');
    }
    drupal_set_message($message);
  }
}

I suspect that something is wrong in the specification of the $batch variable, but I've cross-checked it, and it seems to conform to what the tutorials indicate.

The example in the documentation does not clearly indicate how to pass multiple data points to be called on with the same function.

  'operations' => array(
    array('my_function_1', array($account->id(), 'story')),
    array('my_function_2', array()),
  ),

Interestingly, one tutorial says to pass the data as a single array.

$batch = array(
  'title' => t('Deleting Node...'),
  'operations' => array(
    array(
      '\Drupal\batch_example\DeleteNode::deleteNodeExample',
      array($nids)
    ),
  ),
  'finished' => '\Drupal\batch_example\DeleteNode::deleteNodeExampleFinishedCallback',
);

Another tutorial indicates to pass each data point as a separate element in the operations array.

$operations = [];
foreach ($letters as $letter) {
  $operations[] = ['smackmybatch_process_letter', [$letter]];
}

I'm not entirely sure what this one is specifying.

  'operations' => array(
    array('disc_migrate', array('courses', array('foo' => 'bar'))),
  ),

So, what is the correct way to pass the data that I want to have processed in the batch API?

Answer 1

I have a working batch similar to your example. Here is my code with some comments. I see that you don't use a .batch.inc file, maybe there is the problem, just try using it. I will try to use the same class to see if it works for me.

  public function submitForm(array &$form, FormStateInterface $form_state) {
    //path to file
    $file = drupal_get_path('module', 'diocese_migrate') . '/csv/saint-du-jour.csv';
    //Exploding by ;"0" because the file is malformed
    $lines = explode(';"0"', file_get_contents($file));
    array_pop($lines);

    $operations = [];
    foreach ($lines as $line) {
      $operations[] = ['process_line_saint_du_jour', [$line]];
    }

    $batch = array(
      'title' => t('Migrating Saint of the Day'),
      'operations' => $operations,
      'finished' => 'process_line_saint_du_jour_batch_finished',
      'init_message' => t('Saint of the Day migration is starting.'),
      'progress_message' => t('Processed @current out of @total. Estimated time: @estimate.'),
      'error_message' => t('The migration process has encountered an error.'),
      'file' => drupal_get_path('module', 'diocese_migrate') . '/diocese_migrate.batch.inc',
    );

    batch_set($batch);
  }

diocese_migrate.batch.inc file

<?php
use Drupal\node\Entity\Node;

function process_line_saint_du_jour($line, &$context) {
  $info = str_getcsv($line, ";", '"');

  $title_fr = 'Saint du jour ' . $info[2];

  $node = Node::create([
    // The node entity bundle.
    'type' => 'saint_du_jour',
    'langcode' => 'fr',
    'created' => REQUEST_TIME,
    'changed' => REQUEST_TIME,
    // The user ID.
    'uid' => 1,
    'title' => $title_fr,
  ]);
  $node->save();

  $context['results'][] = $info[3];
  $context['message'] = t('Migrating Saint du jour @day', array('@day' => $info[3]));
}

function process_line_saint_du_jour_batch_finished($success, $results, $operations) {
  // The 'success' parameter means no fatal PHP errors were detected. All
  // other error management should be handled using 'results'.
  if ($success) {
    $message = \Drupal::translation()->formatPlural(
      count($results),
      'One Saint of the Day processed.', '@count Saint of the Day processed.'
    );
  }
  else {
    $message = t('Finished with an error.');
  }
  drupal_set_message($message);
  //$_SESSION['disc_migrate_batch_results'] = $results;
}

Answer 2

Some points about the code you provided:

1. The method (or methods) defined in the 'operations' list should be correctly namespaced; you are using the wrong namespace, which is why see that warning comes and the whole batch is failing.
2. The method (or methods) defined in the 'operations' list should be part of a separate class, as a best practice.
3. The method (or methods) defined in the 'operations' list should be defined as public static methods, as the batch API should not be expected to construct objects, and it should be able to access the method.

General answer to your question:

The correct way to pass data to the batch processing function, is by enclosing them in an array; this is the normalized format that allows you to pass multiple arguments to your function. For example, and for the following batch processing function:

namespace Drupal\mymodule;

class MyBatchClass {
  ...
  public static function myBatchProcessorCallback(
    $someString,
    $uid = 1, 
    &$context) {
    ...
  }  
}

you can add the following operations to your batch definition array:

$batch = array(
  'title' => $this->t('Dummy processing'),
  'operations' => [
    [
      '\Drupal\mymodule\MyBatchClass::myBatchProcessorCallback',   
      ['randomStringA'],
    ],
    [
      '\Drupal\mymodule\MyBatchClass::myBatchProcessorCallback', 
      ['randomStringB', 2],
    ],
  ],
);

Hope this helps, it really does seem that the namespacing is causing all your worries, good luck.

Answer 3

As this was for me a really time consuming issue, I uploaded an example module on my GitHub page: drupal 8 batch import example with interaction.

$batch = [
  'title' => t('Importing animals'),
  'operations' => [],
  'init_message' => t('Import process is starting.'),
  'progress_message' => t('Processed @current out of @total. Estimated time: @estimate.'),
  'error_message' => t('The process has encountered an error.'),
];

foreach($data as $item) {
  $batch['operations'][] = [['\Drupal\batch_import_example\Form\ImportForm', 'importAnimal'], [$item]];
}

batch_set($batch);

Answer 4

Thanks for your answers, everyone. First off, I did not name the function properly, as several have noted.

Second, it turns out that the fourth example has the structure that worked for me. I don't know why it would vary, but whatever.

Here's the submit function from my actual, final module:

  public function submitForm(array &$form, FormStateInterface $form_state) {
    $batch = array(
      'title' => t('test batch'),
      'operations' => array(),
      'finished' => 'Drupal\test_batch\Api\Form\batchCallback',
      'progress_message' => t('Scraped @current out of @total urls.'),
    );
    foreach ( \Drupal\test_scrape\Api\Scraper::loadUrls() as $url ) {
      $batch['operations'][] = [
        '\Drupal\test_scrape\Api\Scraper::batchScrape',
        ['test_scrape', [$url]]
      ];
    }
    batch_set($batch);

And here's the callback its referencing:

src/Api/Scraper.php

  /**
   *  wrapper for batch callback
   */
  public static function batchScrape($type, $operations=[], &$context) {
    foreach ( $operations as $url) {
      self::scrape($url);
      $context['message'] = "Migrated $url<br/>\n"  . $context['message'] ;
    }
  }