2

I've got code that loops through nodes in order to generate processed output.

I've previously done an entity query that provided an array of node IDs. My main loop code looks like this:

foreach ($this->nids as $nid) {
    $node = Node::load($nid);
    $nodeData = $node->toArray();
    $nodeURL = $propertyNode->toUrl()->toString();
    print "Mem usage: ".memory_get_usage(true)."\n<br>";
}

Obviously, the loop shown here is a minimal example to demonstrate the problem; as it stands, it's pretty useless as it doesn't do anything with the data it loads.

However despite this, the loop is eating memory. The debug call to memory_get_usage(true) in the loop gives a bigger number every time, until the script eventually runs out of memory if I feed it enough nodes (about 300 seems to be enough to kill it).

Obviously, I can increase the memory available, but I'm reluctant to do that because I can't seee why the loop would be leaking memory in the first place.

Can anyone please explain why this fairly basic loop is causing a memory leak, and what I can do to prevent it?

A few additional notes:

I've tried adding unset() at the end of the loop to clear the node and data variables; this doesn't have any effect (I wasn't expecting it to, but I tried it anyway).

The nodes in question have about fifty custom fields, including a couple that are paragraphs, which in turn have fields of their own.

My real code passes the node data to a processor class that reformats it for use elsewhere. It builds an array of these objects, but they aren't the cause of the memory leak, hence why I've excluded them from the example code.

I'm currently running on Drupal 8.6; haven't had a chance to look at the 8.7 release from yesterday yet.

  • 5
    It’’s not a leak as such - nodes are statically cached when they’re loaded. Load enough of them and the static cache will fill up the available memory. You can use EntityStorageBase::resetCache to alleviate the problem – Clive May 2 at 15:59
  • Well, sure, you are doing a foreach over a large list that is calling node_load, which in itself makes other calls. Thats a ton of operations. – Kevin May 2 at 15:59
  • 1
    Firstly, just replace Node::load($nid) with Node::loadMultiple($this->nids) and using for foreach. – Jonh May 2 at 16:02
  • 3
    You find an example for @Clive's comment in this question posted recently drupal.stackexchange.com/questions/280633/… – 4k4 May 2 at 20:05
  • Thanks @Clive. See my self-answer; I was able to use your tip to help resolve this. Other comments here helped me get a bit further, but thank you for giving the first useful tip. – Spudley May 7 at 11:37
2

With thanks to the comments made by various people above, I have now optimised this code so that it doesn't run out of memory.

$chunks = array_chunk($this->nids, 100);

foreach ($chunks as $chunk) {
    $nodes = Node::loadMultiple($chunk);

    foreach ($nodes as $node) {
        //do stuff with the node here...
    }
    Drupal::entityTypeManager()->getStorage('node')->resetCache([$chunk]);
    print "Mem usage: ".memory_get_usage(true)."\n<br>";
}

I couldn't simply do loadMultiple for all the nodes at once as it was still using unholy amounts of memory, but chunking it and resetting the cache after each chunk seems to work nicely.

I experimented with different sized chunks, and I found I got the best mix of performance vs memory usage with the chunk size at 100. YMMV.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.