Drupal 8: SQL Insert with 10.000 datasets exceeds 300 MB PHP memory via batch_set()

Question

I have tried to use Drupal's batch_set() function to execute a SQL insert with 10.000 items. It exceeds 300 MB of PHP memory. 512 MB of PHP memory are getting the job done. This is the case, if I use the batch_set() function and if I do not. There is currently no benefit in my use-case, either I use batch_set() or I am not using it. I expect from batch_set() to keep the PHP memory consumption low.

Here is what I am doing. After an user updates the user account, I am updating all nodes for internal purpose. This service method's code is being triggered:

$batch = array(
  'title' => 'Rebuilding user permissions',
  'init_message' => 'Start rebuilding user permissions.',
  'progress_message' => 'Completed @current step of @total.',
  'error_message' => 'Rebuilding user permissions has encountered an error.',
  'operations' => array(
    array(
      [get_class($this), 'updateGrants'],
      [$grantsByNid]
    ),
  ),
);
batch_set($batch);

The $grantsByNid contains around 10.000 items. The method which is triggered via the batch, is this:

/** * @param array $grantsByNid */ public static function updateGrants($grantsByNid) { $query = \Drupal::service('database')->insert('node_access'); $query->fields( ['nid', 'langcode', 'fallback', 'gid', 'realm', 'grant_view', 'grant_update', 'grant_delete'] ); foreach ($grantsByNid as $nid => $grants) { foreach ($grants as $grant) { $query->values([$nid, $grant->langcode, 1, $grant->gid, $grant->realm, $grant->grant_view, $grant->grant_update, $grant->grant_delete]); } } $query->execute(); }

How can I prevent $query->execute(); from causing this error:

( ! ) Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 20480 bytes) in /home/jepster/PhpstormProjects/permissions-by-term-behat/drupal/core/lib/Drupal/Core/Cache/DatabaseCacheTagsChecksum.php on line 108

I have googled a lot and I cannot find out, why Drupal 8's database functionality is using all the memory and why the batch functionality is not preventing this.

Answer 1

Ivan Jaros is correct, right now you just do the work twice.

You're expecting something that is not possible.

Batch saves the operations data in the database, it has to. So you save one huge array only to load it again and store it in a separate table.

A batch makes only sense if you can actually split up the processing into smaller bits and not have all the data in one large array.

Specifically for node access processing, the API for this already exists, no need to re-invent the wheel. See node_access_rebuild(), which can do this either in the current request (not recommended for a large amount of data) or with a batch. And you can see that there is only one batch operation without data, but if you look at it, you can see it is doing a pager query to process the whole list of nodes (there are Drupal sites with millions of nodes).

Answer 2

Providing code example for Berdir's answer here to give you more information.

Read this article https://www.drupal.org/docs/7/api/batch-api/overview

and check the example module here to checkout the proper way of using the batch processing Drupal 8: https://github.com/GiantRobot/csvimport

you will then know how to alter your code to work.

cheers!