1

The batch processed all the items, but instead of showing the finish message, I see the error "Allowed memory size of 536870912 bytes exhausted".

When I debug the code, I notice that Drupal loads each proccessed block on batch finish (ContentEntityBase->__construct). I can't figure out why Drupal is doing that.

Structure of the code:

This is a custom module with form class, and batch functions in the custom_module.module file. On form submit, the module calls the create batch function:

public function submitForm(array &$form, FormStateInterface $form_state) {
  custom_module_make_batch();
}

The custom_module_build_batch function getting ids of custom blocks (4000 or more) and generating the batch:

function custom_module_make_batch()
{

  $batch = [];

  $items = get_blocks_ids();

  $batch = custom_module_generate_batch($items);
  batch_set($batch);
}

function custom_module_generate_batch($items)
{

  $operations = [];

  $operations_groups = array_chunk($items, 50);

  foreach ($operations_groups as $key => $operations_group) {
    $operations[] = [
      'custom_module_batch_op',
      [$operations_group],
    ];
  }

  $batch = [
    'operations' => $operations,
    'finished' => 'custom_module_batch_finished',
    'title' => 'Custom batch',
    'init_message' => 'Batch is starting.',
    'progress_message' => 'Processed @current out of @total parts.',
    'error_message' => 'Batch has encountered an error.',
  ];
  return $batch;
}

function custom_module_batch_op($operations_group, &$context) {

  foreach ($operations_group as $key => $bid) {
  
    $block = \Drupal::service('entity.repository')->loadEntityByUuid('block_content', $bid);

    $block->field_name = $new_value;
    
    $block->save();
  }

}

function custom_module_batch_finished($success, $results, $operations)
{

  $messenger = \Drupal::messenger();
  if ($success) {
    // Here we could do something meaningful with the results.
    // We just display the number of nodes we processed...

    if ($total) {
      $messenger->addMessage(t('@count results processed.', ['@count' => $total]));
    } else {
      $messenger->addMessage(t('There no items for the migration'));
    }

  } else {
    // An error occurred.
    // $operations contains the operations that remained unprocessed.
    $error_operation = reset($operations);
    $messenger->addMessage(
      t(
        'An error occurred while processing @operation with arguments : @args',
        [
          '@operation' => $error_operation[0],
          '@args' => print_r($error_operation[0], TRUE),
        ]
      )
    );
  }
}
3
  • 1
    $new_value is undefined. And better use $block->set('field_MYFIELD', $new_value) to set the value or $block->set('field_MYFIELD', []) to empty it.
    – leymannx
    Mar 19, 2022 at 7:04
  • @leymannx thanks, ok, i will use $block->set() $new_value - it's just an example, it can be array: [ 'value' => 'Some tex', 'format' => 'plain_text', ];
    – Egor Elkin
    Mar 19, 2022 at 9:17
  • 1
    It seems that get_blocks_ids() is loading all the items, which would explain the error message. A batch callback executes the query to get the necessary items. Getting all the items and handing them in batches is not how a batch operation is supposed to work, since batch operations are done to avoid to use all the available memory and avoid time outs caused from PHP taking more time to handle the request than the assigned time.
    – apaderno
    Mar 19, 2022 at 9:40

1 Answer 1

5

Batch operations are used when the number of items to handle is unknown and there could possibly be so much items that either loading all of them would use all the memory PHP make available, or handling them would take more time than the time PHP gives to the script to run. When data is loaded from a database table, even entity data, it's the batch operation callback that loads the data, like this code does, for example.

$batch_builder = (new BatchBuilder())
  ->setTitle(t('Deleting database rows'))
  ->setFinishCallback('mymodule_finished_callback')
  ->addOperation('mymodule_delete_rows', []);

batch_set($batch_builder->toArray());

function mymodule_delete_rows(&$context) {
  $connection = \Drupal::database();

  if (empty($context['sandbox'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_id'] = 0;
    $context['sandbox']['max'] = $connection
      ->query('SELECT COUNT(DISTINCT [id]) FROM {example}')
      ->fetchField();
    $context['results'] = 0;
  }

  $limit = 5;
  $result = $connection
    ->select('example')
    ->condition('id', $context['sandbox']['current_id'], '>')
    ->orderBy('id')
    ->range(0, $limit)
    ->execute();

  foreach ($result as $row) {
    $context['sandbox']['progress']++;
    $context['results']++;
    $context['sandbox']['current_id'] = $row->id;
    
    if (!empty($row->title) && is_numeric($row->title[0])) {
      $connection->delete('example')
        ->condition('id', $row->id)
        ->execute()
    }
  }
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }
}

function mymodule_finished_callback($success, $results, $operations, $elapsed) {
  if ($success) {
    $message = \Drupal::translation()->formatPlural($results, '@count row deleted.', '@count rows deleted.');
  }
  else {
    $message = t('Finished with an error.');
  }
  \Drupal::messenger()->addMessage($message);
}

The relevant values set in the batch operations callback are the following ones:

  • $context['finished'] tells Drupal to stop calling the batch operations callback, when its value is 1 or $context['finished'] isn't set
  • $context['results'] is passed to the callback invoked when the batch operations are done, as its second parameter
  • $context['message'] is a text message displayed in the progress page

I would not build an array containing a value for each handled item, as that would use all the available memory, when the handled items are much.

Reference

7
  • 1
    "The code seems to use all the allowed memory because it first loads all the entities and then passes their IDs to the operation callback. That's not how batch operations should be implemented." - no, the get_blocks_ids() function getting block id's from DB, in order to optimize the speed and memory usage - Please read the topic again, i'm getting "Allowed memory size of 536870912 bytes exhausted" on batch finish, when the script processed all the items Anyway thanks for your thoughts @apaderno
    – Egor Elkin
    Mar 20, 2022 at 10:12
  • Still, the memory gets exhausted also because you are loading the entity IDs and creating an operation every 50 entities; with 4000 entities, that means an array of 80 batch operations instead of a single batch operation.
    – apaderno
    Mar 20, 2022 at 14:56
  • 1
    Same with single batch operation, i tested this variant. And again: the batch process all the items successfully, the error apearch on batch finish page load.
    – Egor Elkin
    Mar 20, 2022 at 15:22
  • If you are creating an array that contains a value for each handled item, that would increase the used memory.
    – apaderno
    Mar 20, 2022 at 16:31
  • 1
    Note that operation and finished callbacks cannot be in a mymodule.install file (which you may try to do because you're generating your batch from hook_install).
    – sonfd
    Mar 29, 2022 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.