How to manage / consume multiple Queue API queues in a module?

Question

For a site I'm running, I'm going to have a few different queues for message sending. For example: high priority email, low priority email, high priority twitter, low priority twitter... etc.

I have set up queues (like "UrgentEmail", "UrgentTwitter") etc. using Drupal's Queue API, but now I need to work on the queue processing (consuming).

I can simply use hook_cron in a module to consume one queue at a time, or all of them, as much as possible, but I'd like to be able to maybe have one queue get hit every couple minutes, while the lower-priority queues would only get it once every half an hour or so...

Is this possible without writing up some of my own cron handlers? I know elysia cron for D6 was good at setting options for module-specific cron, but is there a way I could branch out cron in a single module and run different tasks at different intervals? Or is there another Drupal Way™ to do this?

Answer 1

Drupal does not offer anything like this out of the box.

drupal.org uses Jenkins, which then starts drush scripts. Jenkins is very flexible to configure and has good built-in reporting while drush scripts make it easy to run code inside the context of drupal, without having to worry about authentication (as opposed to accessing it over HTTP, for example).

So, in your case, you could create a number of drush scripts (e.g. a script pro type like twitter/mail + an argument for the urgency) and then configure jenkins to run them at various time intervals.

Answer 2

It's possible to achieve this using Ultimate Cron! module.

This module creates a separate entry for each Queue item. Maybe it will help somebody.

Answer 3

It's not possible to run cron queues at different times, but it's possible to assign to each queue the time spent from Drupal to run it.

function mymodule_cron_queue_info() {
  $queues['mymodule_low_priority'] = array(
    'worker callback' => 'mymodule_low_priority_worker', 
    'time' => 5,
  );

  $queues['mymodule_high_priority'] = array(
    'worker callback' => 'mymodule_high_priority_worker', 
    'time' => 45,
  );

  return $queues;
}

Both the queues will be executed during cron tasks, but assigning less time to the one you consider a low priority queue means that the queue will take more cron tasks to be completed, if all the queues take the same time to be completed. If the queue you consider low priority contains only two items, then it will be completed before other queues that contain more items.