1

I'm working on an Excel exporting feature in a Drupal 8 site.

Obviously, I can load all entites of certain type like this:

$user = \Drupal::currentUser();

$entity_query = \Drupal::service('entity.query')->get('some_type');
$entity_query->sort('created', 'DESC');

$ids = $entity_query->execute();
$entites = $this->storage->loadMultiple($ids);

foreach ($entities as $entity) {
    // build some Excel row with the entity information
    // ...
}

But if you're doing this on a large entity set, you'll be running out of memory very soon. I think the reason why Drupal would run out of memory is the extensive caching involves in the entity query and loading process.

Is there a more memory-efficient way to load an indefinite size of entity set without running out of memory?

(Note: Let's pretend the Excel library is very clever and do not take up memory. That's another issue for another day.)

2 Answers 2

2

Batches allow heavy processing to be spread out over several page requests, ensuring that the processing does not get interrupted because of a PHP timeout, while allowing the user to receive feedback on the progress of the ongoing operations. It also can reduce out of memory situations.

Source: Example: Batch API

You need to use the Batch API. There is no official docs for D8, but you can look at the batch_example module and the CSV Import module.

Good conceptual read, Drupal 8 Batch Processing but a bad code example.

2
  • Not sure if the linked article is such a good read. It does wrong what most articles on Batch API do wrong: Processing everything in just one single batch operation. Whereas this needs to happen in multiple operations each processing a chunk of entities.
    – leymannx
    Jun 9, 2020 at 6:27
  • 2
    There's this csvimport sample module I co-maintain which actually does it right. Though it adds one operation per CSV line. One operation per every 5-10 lines would probably have been fine as well.
    – leymannx
    Jun 9, 2020 at 6:29
2

To reduce the memory footprint you can load single entities and then use resetCache():

$ids = $entity_query->execute();

foreach ($ids as $id) {
  $entity = $this->storage->load($id);
  // build some Excel row with the entity information
  // ...
  // release entity from memory
  $storage->resetCache([$id]);
}
2
  • 2
    The con of just doing this would be that you could potentially run into a PHP timeout, but good to know.
    – No Sssweat
    Jun 9, 2020 at 6:48
  • 1
    @NoSssweat, yes, I know, but the question is specifically about loading entities in a more memory-efficient way and not everyone googling this might be interested in a batch solution, which is in general the better solution though and it is good to have both answers.
    – 4uk4
    Jun 9, 2020 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.