3

I am working on a site with a specific node type that has close to 50 fields, most of which are taxonomy term references, a couple of node references and a few conventional text and integer values. Almost all of them accept multiple values. While indexing the nodes with apachesolr I observe that most of the time in the indexing process is consumed by calls made to entity_load function, almost 3.5 - 4 minutes for loading 200 nodes when checked with timer based logging while the total indexing takes about 50 seconds more (4.5 minutes overall).

Adding a custom 'status back' with hook_apachesolr_entity_info_alter to query and return the node status directly instead of loading it, I was able to negate the entity_load in one instance, however another function apachesolr_index_entity_to_documents() where the drupal node to solr document conversion takes place had entity_load calls consumed more time since they could no longer use the cached result of the status callback.

My aim is to reduce the time taken for indexing to at least half of what it currently which now is completely dependant on the time taken by entity_load. Since entity_load is one of the critical aspects of Drupal 7 around which functions like node_load/node_load_multiple are derived from I'm unable to find a different option to proceed or determine ways to improvise entity_load usage.

Note: The site functions such that these nodes get created almost all the time and none of the fields can be pruned out, negating the option of pre-loading the entities. Also the site uses entity cache module with memcache

I'm hoping to get suggestions on improving the node load time programatically with entity_load or using alternative approach to load the respective entities to bring down my indexing time.

  • I doubt you'll be able to get entity_load() times down, if there was anything more to be had there it would already by in core. The only alternative is direct database queries, which bypass all the metadata in the API (which is why all modules should use the API in the first place) and would have to be updated manually when anything config-related changes. If you can't cache, there's not a lot you can do – Clive Mar 13 '14 at 11:49
  • Are you positive that your MySQL instance is optimized and that your memcache has enough memory and isn't thrashing? – mpdonadio Mar 13 '14 at 12:08
  • @Clive & MPD Thanks for the response and feedback, the memcache stats show about 92% hit and that all subsequent entity_load calls on those nodes take only a few milliseconds make me believe that its holding up to its expectations. Also I'm informed that the MySQL is optimized and tuned for the best performance. Since new nodes keep getting churned in every second and get added to the solr indexing queue I was hoping to get some progress on initial entity load than caching, anyhow will try to check how things look and revert with my observations. – optimusprime619 Mar 14 '14 at 3:20
2

You could very well form an array of the nodes to be indexed after the query execute and do a single entity_load instead of doing a entity_load for each and every node separately and I hope that will reduce the time and below is the patch that could help you

Index: apachesolr/apachesolr.index.inc

--- apachesolr/apachesolr.index.inc (revision 9348) +++ apachesolr/apachesolr.index.inc (working copy) @@ -636,9 +636,21 @@ $query->condition('aie.entity_type', $entity_type); } $query->range(0, $limit); - $records = $query->execute(); - $status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback'); + $records = $query->execute()->fetchAll(); + foreach ($records as $node_record) {
+ $entities[] = $node_record->entity_id;
+ } + $arr_entity = entity_load($entity_type, $entities); + foreach ($records as $record) {
+ $record->status = $record->status && $arr_entity[$record->entity_id]->status; + $rows[] = $record; + } + // Commenting as we have already used the entity_load + /*$status_callbacks = apachesolr_entity_get_callback($entity_type, 'status callback');
foreach ($records as $record) { // Check status and status callbacks before sending to the index if (is_array($status_callbacks)) { @@ -651,7 +663,7 @@ } } $rows[] = $record; - } + }*/ return $rows; }

As you have said most of your fields are taxonomy term it is idle that you check the term_load method and also whether the term_data table is properly indexed.

  • Is there a link to this patch in the apache solr issue queue? – mikeytown2 Mar 27 '14 at 18:41
1

You should first figure out is why is entity_load so slow. What additional hooks are running inside of it that slows it down? An example of a module causing a slow entity_load is the metatag module. I have a patch for it in this issue: https://drupal.org/node/2183203 If you can find what is slowing down your entity_load that would be the most helpful.

If there are no slow performers inside entity_load, make sure that it is loading the entities in bulk and not one at a time.

If all else fails, one of the better looking alternatives is direct database queries at this point if the database layer has already been optimized. I've been working on some code that extends EFQ which I call EntityFieldQueryExtra. I got the idea for it from the apache solr module as apache solr also extends EFQ. For this use case you would only add in the fields that need to be indexed and I would use executeEntityBuild() as that will give you output similar to entity_load; just dpm() what is returned from executeEntityBuild() to find it.

If it is a volume issue (lots of entities to be processed), HTTPRL could be helpful if you want to roll your own solution or if you want to try a module that uses it like Apache Solr Parallel Indexing.

  • Thanks for the response, I'll try to debug for any additional modules that might be slowing down the entity_load calls and try loading multiple entities for any potential performance improvement, BTW awesome sandbox module! – optimusprime619 Mar 14 '14 at 3:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.