Improve performance by using drush functions instead of Drupal module functions

Question

I am trying to improve the import performance of a drupal website we are working on. The site makes use of the queue api to import individual items. Currently this process is very slow. I have hear it mentioned that one way of improving performance is to process items using drush rather than having Drupal do it directly.

What our site does is read the contents of a text file and then add each item to a queue for processing. The queue worker function just takes the information, performs some processing logic and then saves it as a node.

From what I understand we would want to move this queue worker function out of the current Drupal module and into drush.

I have been searching for general background information on how to do this or some introductory tutorials, but have not been able to find anything about using drush to improve performance of resource intensive tasks. I am a little surprised that there is so little information on this given that it is supposed to improve Drupal performance.

Answer 1

The reason you have heard that performance is better under Drush when processing queues is that Drush has a concurrent-execution feature that it uses to run different parts of batch processes in parallel. Running this code on a live server would not necessarily be a good idea, because any "performance gains" that you experienced from using this code would be at the expense of the performance of your front-end. In other words, Drush does not increase the performance of the processing in terms of CPU cycles needed -- it just arranges things so that more of your CPU cycles can be used all at once, so that the task finishes sooner.

I wouldn't even contemplate trying to do this on your live server unless you already have set up a multiple-machine load balancing system, where multiple servers have concurrent access to the same database. (The other alternative would be to run the queue at a time when you don't care about front-end performance -- however, if you have a time when not many people are using the system, you probably don't care when the batch process finishes anyway.) If you do have multiple machines to throw at this problem, you could consider either (a) porting the concurrent-execution functions to a Drupal module (see backend.inc) or (b) making a web service on a separate machine that 'exec's drush queue-run as jonhattan describes.

This would be a little complicated to set up, which is probably why no one has done this yet. A better way to increase performance would be to analyze what is going on in your batch processes that is slow, and see if you can find any ways to optimize the code so that it uses fewer CPU cycles. This would result in an actual performance increase, which would be preferable to a complex system that processed a slow batch process efficiently.

Answer 2

Queues are defined within Drupal modules, and you can use drush to run those queues:

drush @site queue-list
drush @site queue-run queue_name

Changing the interface to the import code (queue, batch api or a drush command) won't affect drastically the import time.

I'd review the import code to try and optimize it. For example by caching (in memory) any data you may be loading recurringly from the database.