1

I am trying to improve the import performance of a drupal website we are working on. The site makes use of the queue api to import individual items. Currently this process is very slow. I have hear it mentioned that one way of improving performance is to process items using drush rather than having Drupal do it directly.

What our site does is read the contents of a text file and then add each item to a queue for processing. The queue worker function just takes the information, performs some processing logic and then saves it as a node.

From what I understand we would want to move this queue worker function out of the current Drupal module and into drush.

I have been searching for general background information on how to do this or some introductory tutorials, but have not been able to find anything about using drush to improve performance of resource intensive tasks. I am a little surprised that there is so little information on this given that it is supposed to improve Drupal performance.

Improve this question
1
  • You could create a drush command for your module, drush.ws/docs/commands.html. But to me it sounds like the code itself is slow. Executing it from drush won't gain much performance. If you update your question with the code that is being executed maybe you could get pointers on how to optimize it.
    – Pontus Nilsson
    Jun 19, 2014 at 7:31

2 Answers 2

Reset to default
2

The reason you have heard that performance is better under Drush when processing queues is that Drush has a concurrent-execution feature that it uses to run different parts of batch processes in parallel. Running this code on a live server would not necessarily be a good idea, because any "performance gains" that you experienced from using this code would be at the expense of the performance of your front-end. In other words, Drush does not increase the performance of the processing in terms of CPU cycles needed -- it just arranges things so that more of your CPU cycles can be used all at once, so that the task finishes sooner.

I wouldn't even contemplate trying to do this on your live server unless you already have set up a multiple-machine load balancing system, where multiple servers have concurrent access to the same database. (The other alternative would be to run the queue at a time when you don't care about front-end performance -- however, if you have a time when not many people are using the system, you probably don't care when the batch process finishes anyway.) If you do have multiple machines to throw at this problem, you could consider either (a) porting the concurrent-execution functions to a Drupal module (see backend.inc) or (b) making a web service on a separate machine that 'exec's drush queue-run as jonhattan describes.

This would be a little complicated to set up, which is probably why no one has done this yet. A better way to increase performance would be to analyze what is going on in your batch processes that is slow, and see if you can find any ways to optimize the code so that it uses fewer CPU cycles. This would result in an actual performance increase, which would be preferable to a complex system that processed a slow batch process efficiently.

Improve this answer
2

Queues are defined within Drupal modules, and you can use drush to run those queues:

drush @site queue-list
drush @site queue-run queue_name

Changing the interface to the import code (queue, batch api or a drush command) won't affect drastically the import time.

I'd review the import code to try and optimize it. For example by caching (in memory) any data you may be loading recurringly from the database.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.