Do I need a cron task for processing a queue?

Question

I've got a task that takes about 45 minutes to complete and needs to happen every day (synching users to several external database, etc).

To handle the work, I've set up a cron queue with hook_cron_queue_info() as follows:

function mymodule_cron_queue_info() {
  $queues = array();
  $queues['update_users_queue'] = array(
    'worker callback' => '_mymodule_process_user_queue_item',
    'time' => 120,
  );
  return $queues;
}

I fill the queue using this function:

function mymodule_queue_all_users_for_synching() {
  //...query for users...

  $queue = DrupalQueue::get('update_users_queue');
  foreach($users as $user) {
    $queue->createItem($user);
  }
}

The queue-filling function gets called as a cron task. I use Elysia Cron, so my implementation of hook_cronapi() is:

function mymodule_cronapi($op, $job = NULL) {
  $items = array();
  $items['queue_users_for_synch'] = array(
    'description' => 'Queue all user accounts for synching.',
    'rule' => '0 3 * * *', // Run this job every day at 3am.
    'callback' => 'mymodule_queue_all_users_for_synching',
  );
  return $items;
}

The worker function for each queue item, defined in mymodule_cron_queue_info is like:

function _mymodule_process_user_queue_item($item) {
  //...synchronize user ($item)...
}

My question is, when is cron actually going to start processing the queue?

Say I fill the queue every day at 3am, and want to process it 120 seconds out of every 30 minutes until it's done--do I need to create another cron task?

Answer 1

When Drupal runs cron tasks, it automatically handles any cron queue defined from modules, in drupal_cron_run(); first hook_cron() implementations are invoked, and then the cron queues are emptied.

Inmplementing hook_cronapi(), you can add an entry for another function handling the cron queue of your module.

function mymodule_cronapi($op, $job = NULL) {
  $items = array();

  $items['queue_users_for_synch'] = array(
    'description' => 'Queue all user accounts for synching.',
    'rule' => '0 3 * * *', // Run this job every day at 3am.
    'callback' => 'mymodule_queue_all_users_for_synching',
  );

  $items['clean_queue'] = array(
    'description' => 'Clean the queue for the user synching.',
    'rule' => '0 4 * * *', // Run this job every day at 4 AM.
    'callback' => 'mymodule_clean_queue',
  );

  return $items;
}

function mymodule_clean_queue() {
  $queues = module_invoke('mymodule', 'cron_queue_info');
  drupal_alter('cron_queue_info', $queues);

  // Make sure every queue exists. There is no harm in trying to recreate an
  // existing queue.
  foreach ($queues as $queue_name => $info) {
    DrupalQueue::get($queue_name)->createQueue();
  }

  foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }
}

The alternative is letting Drupal handle the cron queue for you, but that happens when Drupal cron tasks are executed. If you want to empty the cron queue of your module more frequently, you can only add a new cron task handled by the Elysia Cron module.

The Elysia Cron module handles the cron queues in elysia_cron_run(); this function is being invoked from elysia_cron_cron() (an implementation of hook_cron()), drush_elysia_cron_run_wrapper() (a Drush command callback), and from its own cron.php. If you followed the instructions in the INSTALL.txt file (in particular in "STEP B: CHANGE SYSTEM CRONTAB (OPTIONAL)"), and replaced any invocation of http://example.com/cron.php with http://example.com/sites/all/modules/elysia_cron/cron.php, the Elysia Cron module should be already handling the cron queues. The code I suggested could be used to speed up handling the cron queues used from your module, if there is effectively the need of doing so.

// This code is part of the code executed from modules/elysia_cron/cron.php.
define('DRUPAL_ROOT', getcwd());

include_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_override_server_variables(array(
  'SCRIPT_NAME' => '/cron.php',
));
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

if (!isset($_GET['cron_key']) || variable_get('cron_key', 'drupal') != $_GET['cron_key']) {
  watchdog('cron', 'Cron could not run because an invalid key was used.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
elseif (variable_get('maintenance_mode', 0)) {
  watchdog('cron', 'Cron could not run because the site is in maintenance mode.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
else {
  if (function_exists('elysia_cron_run')) {
    elysia_cron_run();
  }
  else {
    drupal_cron_run();
  }
}

Answer 2

The queue will be populated via the Elysia cronapi hook at the set time.

However, the queue will be processed whenever the standard Drupal cron run occurs.

See this worker callback processing snippet at the end of core: drupal_cron_run

 foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }

Answer 3

as stated above when using Elysia Cron your queues do not get processed.

you (and drupal) have no access to queues that would otherwise run on drupal_run_cron

the workaround is to create a custom cron task - (this will be visible to elysia cron) to either process all queues or one you want and invoke queue processing there. i.e:

function mymodule_cron() {
// below is copied from drupal_cron_run() in common.inc

// Grab the defined cron queues.
$queues = module_invoke_all('cron_queue_info');
drupal_alter('cron_queue_info', $queues);

//if you want to target only one queue you can remove 'foreach'
and your $info = $queues['your_queue'];

  foreach ($queues as $queue_name => $info) {
    if (!empty($info['skip on cron'])) {
      // Do not run if queue wants to skip.
      continue;
    }
    $callback = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
     while (time() < $end && ($item = $queue->claimItem())) {
      try {
        call_user_func($callback, $item->data);
        $queue->deleteItem($item);
      }
      catch (Exception $e) {
        // In case of exception log it and leave the item in the queue
        // to be processed again later.
        watchdog_exception('cron', $e);
      }
    }
  }
}

now queues processing can be controlled by ElysiaCron

Answer 4

I don't use Elysia, but my solution have always been something like this:

function mymodule_cron() {
  $queue = DrupalQueue::get('mymoudule_queue');
  $queue->createQueue();
  $item = $queue->claimItem(300);

  if (!empty($item->data)) {

    // Do the work.

    if ($sucess) {
      $queue->deleteItem($item);
      watchdog('mymodule', 'It worked.');
    }
    else {
      watchdog('mymodule', 'It did not work!', array(), WATCHDOG_ALERT);
    }
  }
}

It only handles one item, for each cron run. Maybe you want to change that.

Answer 5

I've also been trying to wrap my head around this, as I'm using the Queue API for the first time along with Elysia cron. Upon closer inspection you can see that Elysia cron executes queue items when the function elysia_cron_run is called. See this snippet from line 1044 within the file elysia_cron.module:

if (EC_DRUPAL_VERSION >= 7) {
  // D7 Queue processing
  foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }
}

This helped demystify queue processing for me when using Elysia cron.