I am working on a site with a specific node type that has close to 50 fields, most of which are taxonomy term references, a couple of node references and a few conventional text and integer values. Almost all of them accept multiple values.
While indexing the nodes with apachesolr I observe that most of the time in the indexing process is consumed by calls made to
entity_load function, almost 3.5 - 4 minutes for loading 200 nodes when checked with timer based logging while the total indexing takes about 50 seconds more (4.5 minutes overall).
Adding a custom 'status back' with
hook_apachesolr_entity_info_alter to query and return the node status directly instead of loading it, I was able to negate the
entity_load in one instance, however another function
apachesolr_index_entity_to_documents() where the drupal node to solr document conversion takes place had
entity_load calls consumed more time since they could no longer use the cached result of the status callback.
My aim is to reduce the time taken for indexing to at least half of what it currently which now is completely dependant on the time taken by
entity_load is one of the critical aspects of Drupal 7 around which functions like
node_load_multiple are derived from I'm unable to find a different option to proceed or determine ways to improvise entity_load usage.
Note: The site functions such that these nodes get created almost all the time and none of the fields can be pruned out, negating the option of pre-loading the entities. Also the site uses entity cache module with memcache
I'm hoping to get suggestions on improving the node load time programatically with entity_load or using alternative approach to load the respective entities to bring down my indexing time.