7

I have always known that loading multiple nodes at the same time is faster than loading them one by one.

What I wasn't aware of, is the huge performance difference between the two ways of loading nodes.

I made an example to see the difference. Here's my result:

Load 1000 nodes one by one:

$start_time = microtime(TRUE);
foreach (range(1, 1000) as $nid) {
  $node = \Drupal::entityTypeManager()->getStorage('node')->load($nid);
  if ($node) {
    print_r($node->id());
  }
}
$end_time = microtime(TRUE);
print_r($end_time - $start_time);

Execution time: 10.7266 seconds.

Load 1000 nodes at the same time:

$start_time = microtime(TRUE);
$nids = [];
foreach (range(1, 1000) as $nid) {
  $nids[] = $nid;
}

$nodes = \Drupal::entityTypeManager()->getStorage('node')->loadMultiple($nids);

foreach ($nodes as $node) {
  if ($node) {
    print_r($node->id());
  }
}
$end_time = microtime(TRUE);
print_r($end_time - $start_time);

Execution time: 1.1664 second.

How can loadMultiple be 9.2 times faster than loading the nodes one by one?

Is it the connection between Drupal and the database? What other factors are slowing down the process when loading the nodes one by one?

I have cleared the cache after every request to avoid the loaded nodes being cached.

I used the microtime function to calculate the exact execution times.

6
  • I really like your efforts on this site and I'm really happy about your activity here so please don't understand this wrong, I definitely don't want to scare you off. But while this question involves Drupal functions it's actually a question about PHP in general. It's up to you then to have a look a these functions and understand the bare PHP code which is being executed in there. If that's unclear you are better off asking on SO, in my opinion.
    – leymannx
    Jul 18, 2019 at 8:47
  • I think it’s Drupal related - the question is on how these two Drupal functions operate in the backend.
    – Jaypan
    Jul 18, 2019 at 8:50
  • Hi @leymannx, first of all thank you for the nice comment, I really appreciate. I just have a question. Imagine I'm going to ask this question on SO, I would have to tag the question drupal because PHP experts that doesn't know anything about Drupal would think \Drupal::entityTypeManager() that's definetely not a PHP function I know... So while it might be related to PHP code, I still think it's on topic here, because it's about some of Drupal's core functions and how they operate.
    – user72672
    Jul 18, 2019 at 8:54
  • Well, I'm probably a bit more strict when it comes to flagging than others. Since I like how well-curated this site is and want to keep it that way. Asking on SO and tagging it drupal doesn't automatically make a question on-topic for here. Experts on SO probably answer: Well, what's happening in that function, have you had a look, what exactly is unclear about it? Maybe it's even more obvious to them that a foreach always takes longer than loading one large chunk of data at once.
    – leymannx
    Jul 18, 2019 at 9:24
  • 3
    @leymannx This is definitely a Drupal-specific question, and there's actually quite a bit more to it than the current question/answer have gone into. For example, the premise of the multiple load being 20x quicker simply isn't true if relevant static caches have been primed. That, and other related, architectural decisions are, while not unique to or invented by Drupal, the result of specific decisions made by the core devs when this was built, so we should expect to cover it here IMO
    – Clive
    Jul 18, 2019 at 11:19

2 Answers 2

7

It's likely that when you are loading individual nodes, you are loading the entity type manager, then for each node retrieve the storage, do a query to get the node data, and build the node. With the multiple process, some of the DB queries are likely combined, and the processing will be in a batch rather than having to run through the code multiple times.

3
  • 8
    Side note: EntityTypeManager won't be initiated multiple times. It's a service and therefore always the same instance will be used. The main performance difference between the two is definitely the db query(ies).
    – ssibal
    Jul 18, 2019 at 11:56
  • 1
    Did you notice that load() is a wrapper function for loadMutliple() in EntityStorageBase?
    – DiDebru
    Jul 26, 2019 at 11:33
  • Yes but it will be called one thousand times - once for each node.
    – Jaypan
    Jul 26, 2019 at 12:19
2

This actually has nothing to do with the storage or entity type manager itself.

However, it is true that loadMultiple is faster in this case.

The slowness comes from \Drupal\Core\Entity\Sql\SqlContentEntityStorage::getFromStorage to be exact.

As a matter of fact, even if you do call load, it will call loadMultiple.

For the reason why loading entities individually is slower is because how the data is queried from database. In most usual scenario entity has base table, data table and possible revision tables. With node, you often have also extra fields, of which have separate table as well.

So as bare minimum on every load, you will be querying node, node_field_data, node_revision and node_field_revision.

When you load multiple entities, under the hood these tables are called only once including all entity IDs. Total of 4 queries.

When you load individually entities, under the hood these tables are also called for every entity. That's total of 4000 queries in your example.

If you have 20 extra fields for node, that would mean 44 queries vs 44000 queries.

Here's snippet you can utilize to see the difference in action.

<?php

$entityType = 'node';
$storage = \Drupal::entityTypeManager()->getStorage($entityType);
$ids = $storage->getQuery()->range(0, 1000)->execute();
$storage->resetCache();

$startTime = microtime(TRUE);
foreach ($ids as $id) {
  $entity = $storage->load($id);
  unset($entity);
}
$endTime = microtime(TRUE);
$runTime = $endTime - $startTime;
echo "Individual: {$runTime} seconds" . PHP_EOL;

$storage->resetCache();

$startTime = microtime(TRUE);
$entities = $storage->loadMultiple($ids);
unset($entities);
$endTime = microtime(TRUE);
$runTime = $endTime - $startTime;
echo "Multiple: {$runTime} seconds" . PHP_EOL;
1
  • So as bare minimum on every load, you will be querying node, node_field_data, node_revision and node_field_revision. Not after the first load, as the node will be cached after that point.
    – Jaypan
    Apr 12 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.