4

I'm working on a Drupal 8 custom module to scrape pages from a site for migration. Since we're dealing with a large number of pages, I wanted to set up batch processing, to deal with timeouts.

I looked at two tutorials, this one, and one here. However, when I ran my batch, it didn't work as expected. The progress bar immediately filled up to all 71 items, and there were no imported pages, but a number of errors in the error log.

To rule out confounding factors, I created a simple module for testing the batch api (I didn't find the batch represented in the D8 examples module). It exhibits the same behavior: it immediately processes all items, and there are no successes, just errors. This an the example error:

 Warning: call_user_func_array() expects parameter 1 to be a valid callback, class '\Drupal\batch_test\BatchTestForm' not found in _batch_process() (line 252 of C:\wamp64\www\D8\core\includes\batch.inc)  
`#0` C:\wamp64\www\D8\core\includes\bootstrap.inc(566): _drupal_error_handler_real(2, 'call_user_func_...', 'C:\\wamp64\\www\\p...', 252, Array)  
`#1` [internal function]: _drupal_error_handler(2, 'call_user_func_...', 'C:\\wamp64\\www\\p...', 252, Array)  
`#2` C:\wamp64\www\D8\core\includes\batch.inc(252): call_user_func_array('\\Drupal\\batch_t...', NULL)  
`#3` C:\wamp64\www\D8\core\includes\batch.inc(95): _batch_process()  
`#4` C:\wamp64\www\D8\core\includes\batch.inc(77): _batch_do()  
`#5` C:\wamp64\www\D8\core\modules\system\src\Controller\BatchController.php(55): _batch_page(Object(Symfony\Component\HttpFoundation\Request))  
`#6` [internal function]: Drupal\system\Controller\BatchController->batchPage(Object(Symfony\Component\HttpFoundation\Request))  
`#7` C:\wamp64\www\D8\core\lib\Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array(Array, Array)  
`#8` C:\wamp64\www\D8\core\lib\Drupal\Core\Render\Renderer.php(574): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber\{closure}()

The following is the form that I'm using. Both of the tutorials indicate that calling _batch_process() is not necessary when calling batch_set() in a form submit.

namespace Drupal\batch_test\Form;
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Form\FormBase;

class BatchTestForm extends FormBase {

  public function getFormId() {
    return 'batch_test_admin_form';
  }

  public function buildForm(array $form, FormStateInterface $form_state) {
    $form['submit'] = [
      '#type' => 'submit',
      '#value' => $this->t('Test Batch'),
      '#button_type' => 'primary',
    ];
    return $form;
  }

  public function submitForm(array &$form, FormStateInterface $form_state) {
    $batch = array(
      'title' => t('Test Batch'),
      'operations' => [],
      'finished' => 'Drupal\BatchTest\Form\batchCallback',
      'progress_message' => t('Processed @current out of @total.'),
    );

    for ( $i = 0; $i <= mt_rand(10,100); $i++ ) {
      $batch['operations'][] = ['\Drupal\batch_test\BatchTestForm::process',$i];
    }

    batch_set($batch);
  }

  public function process($input, &$context) {
    // log a random string
    \Drupal::logger('batch_test')->notice(base64_encode(random_bytes(10)));
    sleep(mt_rand(1,100));
  }

  public function batchCallback($success, $results, $operations) {
    if ($success) {
      $message = \Drupal::translation()->formatPlural(
        count($results),
        'One page scraped.', '@count pages scraped.'
      );
    }
    else {
      $message = t('Finished with an error.');
    }
    drupal_set_message($message);
  }
}

I suspect that something is wrong in the specification of the $batch variable, but I've cross-checked it, and it seems to conform to what the tutorials indicate.

The example in the documentation does not clearly indicate how to pass multiple data points to be called on with the same function.

  'operations' => array(
    array('my_function_1', array($account->id(), 'story')),
    array('my_function_2', array()),
  ),

Interestingly, one tutorial says to pass the data as a single array.

$batch = array(
  'title' => t('Deleting Node...'),
  'operations' => array(
    array(
      '\Drupal\batch_example\DeleteNode::deleteNodeExample',
      array($nids)
    ),
  ),
  'finished' => '\Drupal\batch_example\DeleteNode::deleteNodeExampleFinishedCallback',
);

Another tutorial indicates to pass each data point as a separate element in the operations array.

$operations = [];
foreach ($letters as $letter) {
  $operations[] = ['smackmybatch_process_letter', [$letter]];
}

I'm not entirely sure what this one is specifying.

  'operations' => array(
    array('disc_migrate', array('courses', array('foo' => 'bar'))),
  ),

So, what is the correct way to pass the data that I want to have processed in the batch API?

  • According to your namespace declaration $batch['operations'][] = ['\Drupal\batch_test\BatchTestForm::process',$i]; should be $batch['operations'][] = ['\Drupal\batch_test\Form\BatchTestForm::process',$i];; which would explain the warning your get first. – Stefanos Petrakis Oct 4 '17 at 15:20
4
+25

I have a working batch similar to your example. Here is my code with some comments. I see that you don't use a .batch.inc file, maybe there is the problem, just try using it. I will try to use the same class to see if it works for me.

  public function submitForm(array &$form, FormStateInterface $form_state) {
    //path to file
    $file = drupal_get_path('module', 'diocese_migrate') . '/csv/saint-du-jour.csv';
    //Exploding by ;"0" because the file is malformed
    $lines = explode(';"0"', file_get_contents($file));
    array_pop($lines);

    $operations = [];
    foreach ($lines as $line) {
      $operations[] = ['process_line_saint_du_jour', [$line]];
    }

    $batch = array(
      'title' => t('Migrating Saint of the Day'),
      'operations' => $operations,
      'finished' => 'process_line_saint_du_jour_batch_finished',
      'init_message' => t('Saint of the Day migration is starting.'),
      'progress_message' => t('Processed @current out of @total. Estimated time: @estimate.'),
      'error_message' => t('The migration process has encountered an error.'),
      'file' => drupal_get_path('module', 'diocese_migrate') . '/diocese_migrate.batch.inc',
    );

    batch_set($batch);
  }

diocese_migrate.batch.inc file

<?php
use Drupal\node\Entity\Node;

function process_line_saint_du_jour($line, &$context) {
  $info = str_getcsv($line, ";", '"');

  $title_fr = 'Saint du jour ' . $info[2];

  $node = Node::create([
    // The node entity bundle.
    'type' => 'saint_du_jour',
    'langcode' => 'fr',
    'created' => REQUEST_TIME,
    'changed' => REQUEST_TIME,
    // The user ID.
    'uid' => 1,
    'title' => $title_fr,
  ]);
  $node->save();

  $context['results'][] = $info[3];
  $context['message'] = t('Migrating Saint du jour @day', array('@day' => $info[3]));
}

function process_line_saint_du_jour_batch_finished($success, $results, $operations) {
  // The 'success' parameter means no fatal PHP errors were detected. All
  // other error management should be handled using 'results'.
  if ($success) {
    $message = \Drupal::translation()->formatPlural(
      count($results),
      'One Saint of the Day processed.', '@count Saint of the Day processed.'
    );
  }
  else {
    $message = t('Finished with an error.');
  }
  drupal_set_message($message);
  //$_SESSION['disc_migrate_batch_results'] = $results;
}
  • Can you think of a way t improve your answer by trying to answer the question at the very end of the question itself? – Pierre.Vriens Oct 7 '17 at 20:55
  • @Pierre.Vriens it seems that the user found the answer using the class version :-) I'm prefer this approach using a file with functions, What version do you think is better? Using a class with static functions or using a file with functions? – Adrian Cid Almaguer Oct 12 '17 at 14:14
  • Adrian, not sure which one is "better", but my preference is similar to yours! PS: a shame the bounty hasn't been granted (yet) ... to "your" answer (a much better way to say "thanks", no?). Nevertheless, looking at the timeline of this question it looks like in about 2 hours from now your answer will receive 50% of that bounty anyway ... according to the Rules (oeps: rules) of SE-bounties ... – Pierre.Vriens Oct 12 '17 at 14:23
  • @Pierre.Vriens it seems that is not for me, the point has gone with no reward: "If the bounty was started by the question owner, and the question owner accepts an answer posted during the bounty period, and the bounty expires without an explicit award then we assume the bounty owner liked the answer they accepted and award it the full bounty amount at the time of bounty expiration." – Adrian Cid Almaguer Oct 12 '17 at 15:32
  • The bounty started on Oct 4, at 15:00 UTC. Add to that exactly 7 days + 1 day grace period. If then the bounty expired (and not granted to anybody), it takes about 1 more hour for some "next batch run" to execute. That run will then pick the first answer posted AFTER the bounty, which has a voting balance of at least 2 ... and grant 50% of the bounty amount anyway ... Check the timeline again, to see which answer will get that 50% in the coming minutes or so, wanna bet? If it's not your answer, then it is a bug in this process, trust me ... PS: some SE users don't like my SE-knowledge ... – Pierre.Vriens Oct 12 '17 at 16:05
0

Some points about the code you provided:

  1. The method (or methods) defined in the 'operations' list should be correctly namespaced; you are using the wrong namespace, which is why see that warning comes and the whole batch is failing.
  2. The method (or methods) defined in the 'operations' list should be part of a separate class, as a best practice.
  3. The method (or methods) defined in the 'operations' list should be defined as public static methods, as the batch API should not be expected to construct objects, and it should be able to access the method.

General answer to your question:

The correct way to pass data to the batch processing function, is by enclosing them in an array; this is the normalized format that allows you to pass multiple arguments to your function. For example, and for the following batch processing function:

namespace Drupal\mymodule;

class MyBatchClass {
  ...
  public static function myBatchProcessorCallback(
    $someString,
    $uid = 1, 
    &$context) {
    ...
  }  
}

you can add the following operations to your batch definition array:

$batch = array(
  'title' => $this->t('Dummy processing'),
  'operations' => [
    [
      '\Drupal\mymodule\MyBatchClass::myBatchProcessorCallback',   
      ['randomStringA'],
    ],
    [
      '\Drupal\mymodule\MyBatchClass::myBatchProcessorCallback', 
      ['randomStringB', 2],
    ],
  ],
);

Hope this helps, it really does seem that the namespacing is causing all your worries, good luck.

0

Thanks for your answers, everyone. First off, I did not name the function properly, as several have noted.

Second, it turns out that the fourth example has the structure that worked for me. I don't know why it would vary, but whatever.

Here's the submit function from my actual, final module:

  public function submitForm(array &$form, FormStateInterface $form_state) {
    $batch = array(
      'title' => t('test batch'),
      'operations' => array(),
      'finished' => 'Drupal\test_batch\Api\Form\batchCallback',
      'progress_message' => t('Scraped @current out of @total urls.'),
    );
    foreach ( \Drupal\test_scrape\Api\Scraper::loadUrls() as $url ) {
      $batch['operations'][] = [
        '\Drupal\test_scrape\Api\Scraper::batchScrape',
        ['test_scrape', [$url]]
      ];
    }
    batch_set($batch);

And here's the callback its referencing:

src/Api/Scraper.php

  /**
   *  wrapper for batch callback
   */
  public static function batchScrape($type, $operations=[], &$context) {
    foreach ( $operations as $url) {
      self::scrape($url);
      $context['message'] = "Migrated $url<br/>\n"  . $context['message'] ;
    }
  }
0

As this was for me a really time consuming issue, I uploaded an example module on my GitHub page: drupal 8 batch import example with interaction.

$batch = [
  'title' => t('Importing animals'),
  'operations' => [],
  'init_message' => t('Import process is starting.'),
  'progress_message' => t('Processed @current out of @total. Estimated time: @estimate.'),
  'error_message' => t('The process has encountered an error.'),
];

foreach($data as $item) {
  $batch['operations'][] = [['\Drupal\batch_import_example\Form\ImportForm', 'importAnimal'], [$item]];
}

batch_set($batch);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.