Minimizing the time taken for entity_load while using for apachesolr indexing

Question

I am working on a site with a specific node type that has close to 50 fields, most of which are taxonomy term references, a couple of node references and a few conventional text and integer values. Almost all of them accept multiple values. While indexing the nodes with apachesolr I observe that most of the time in the indexing process is consumed by calls made to entity_load function, almost 3.5 - 4 minutes for loading 200 nodes when checked with timer based logging while the total indexing takes about 50 seconds more (4.5 minutes overall).

Adding a custom 'status back' with hook_apachesolr_entity_info_alter to query and return the node status directly instead of loading it, I was able to negate the entity_load in one instance, however another function apachesolr_index_entity_to_documents() where the drupal node to solr document conversion takes place had entity_load calls consumed more time since they could no longer use the cached result of the status callback.

My aim is to reduce the time taken for indexing to at least half of what it currently which now is completely dependant on the time taken by entity_load. Since entity_load is one of the critical aspects of Drupal 7 around which functions like node_load/node_load_multiple are derived from I'm unable to find a different option to proceed or determine ways to improvise entity_load usage.

Note: The site functions such that these nodes get created almost all the time and none of the fields can be pruned out, negating the option of pre-loading the entities. Also the site uses entity cache module with memcache

I'm hoping to get suggestions on improving the node load time programatically with entity_load or using alternative approach to load the respective entities to bring down my indexing time.

Answer 1

You could very well form an array of the nodes to be indexed after the query execute and do a single entity_load instead of doing a entity_load for each and every node separately and I hope that will reduce the time and below is the patch that could help you

Index: apachesolr/apachesolr.index.inc

--- apachesolr/apachesolr.index.inc     (revision 9348)
+++ apachesolr/apachesolr.index.inc     (working copy)
@@ -636,9 +636,21 @@
     $query->condition('aie.entity_type', $entity_type);
   }
   $query->range(0, $limit);
-  $records = $query->execute();
-  $status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback');
+  $records = $query->execute()->fetchAll();
+  foreach ($records as $node_record) {

+    $entities[] = $node_record->entity_id;

+  }
+  $arr_entity = entity_load($entity_type, $entities);
+  foreach ($records as $record) {

+    $record->status = $record->status && $arr_entity[$record->entity_id]->status;
+    $rows[] = $record;
+  }
+  // Commenting as we have already used the entity_load
+  /*$status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback');

   foreach ($records as $record) {
     // Check status and status callbacks before sending to the index
     if (is_array($status_callbacks)) {
@@ -651,7 +663,7 @@
       }
     }
     $rows[] = $record;
-  }
+  }*/
   return $rows;
 }

As you have said most of your fields are taxonomy term it is idle that you check the term_load method and also whether the term_data table is properly indexed.

Answer 2

You should first figure out is why is entity_load so slow. What additional hooks are running inside of it that slows it down? An example of a module causing a slow entity_load is the metatag module. I have a patch for it in this issue: https://drupal.org/node/2183203 If you can find what is slowing down your entity_load that would be the most helpful.

If there are no slow performers inside entity_load, make sure that it is loading the entities in bulk and not one at a time.

If all else fails, one of the better looking alternatives is direct database queries at this point if the database layer has already been optimized. I've been working on some code that extends EFQ which I call EntityFieldQueryExtra. I got the idea for it from the apache solr module as apache solr also extends EFQ. For this use case you would only add in the fields that need to be indexed and I would use executeEntityBuild() as that will give you output similar to entity_load; just dpm() what is returned from executeEntityBuild() to find it.

If it is a volume issue (lots of entities to be processed), HTTPRL could be helpful if you want to roll your own solution or if you want to try a module that uses it like Apache Solr Parallel Indexing.