I am working on a site with a specific node type that has close to 50 fields, most of which are taxonomy term references, a couple of node references and a few conventional text and integer values. Almost all of them accept multiple values.
While indexing the nodes with apachesolr I observe that most of the time in the indexing process is consumed by calls made to entity_load
function, almost 3.5 - 4 minutes for loading 200 nodes when checked with timer based logging while the total indexing takes about 50 seconds more (4.5 minutes overall).
Adding a custom 'status back' with hook_apachesolr_entity_info_alter
to query and return the node status directly instead of loading it, I was able to negate the entity_load
in one instance, however another function apachesolr_index_entity_to_documents()
where the drupal node to solr document conversion takes place had entity_load
calls consumed more time since they could no longer use the cached result of the status callback.
My aim is to reduce the time taken for indexing to at least half of what it currently which now is completely dependant on the time taken by entity_load
. Since entity_load
is one of the critical aspects of Drupal 7 around which functions like node_load
/node_load_multiple
are derived from I'm unable to find a different option to proceed or determine ways to improvise entity_load usage.
Note: The site functions such that these nodes get created almost all the time and none of the fields can be pruned out, negating the option of pre-loading the entities. Also the site uses entity cache module with memcache
I'm hoping to get suggestions on improving the node load time programatically with entity_load or using alternative approach to load the respective entities to bring down my indexing time.
entity_load()
times down, if there was anything more to be had there it would already by in core. The only alternative is direct database queries, which bypass all the metadata in the API (which is why all modules should use the API in the first place) and would have to be updated manually when anything config-related changes. If you can't cache, there's not a lot you can do – Clive♦ Mar 13 '14 at 11:49