Take the 2-minute tour ×
Drupal Answers is a question and answer site for Drupal developers and administrators. It's 100% free, no registration required.

I've got a task that takes about 45 minutes to complete and needs to happen every day (synching users to several external database, etc).

To handle the work, I've set up a cron queue with hook_cron_queue_info() as follows:

function mymodule_cron_queue_info() {
  $queues = array();
  $queues['update_users_queue'] = array(
    'worker callback' => '_mymodule_process_user_queue_item',
    'time' => 120,
  );
  return $queues;
}

I fill the queue using this function:

function mymodule_queue_all_users_for_synching() {
  //...query for users...

  $queue = DrupalQueue::get('update_users_queue');
  foreach($users as $user) {
    $queue->createItem($user);
  }
}

The queue-filling function gets called as a cron task. I use Elysia Cron, so my implementation of hook_cronapi() is:

function mymodule_cronapi($op, $job = NULL) {
  $items = array();
  $items['queue_users_for_synch'] = array(
    'description' => 'Queue all user accounts for synching.',
    'rule' => '0 3 * * *', // Run this job every day at 3am.
    'callback' => 'mymodule_queue_all_users_for_synching',
  );
  return $items;
}

The worker function for each queue item, defined in mymodule_cron_queue_info is like:

function _mymodule_process_user_queue_item($item) {
  //...synchronize user ($item)...
}

My question is, when is cron actually going to start processing the queue?

Say I fill the queue every day at 3am, and want to process it 120 seconds out of every 30 minutes until it's done--do I need to create another cron task?

share|improve this question
    
I should mention I'm using Drupal 7. –  joe_flash Jan 6 '13 at 1:53
    
I'm curious as well about this questions. Would like to hear yes or no. We use queue API heavily in one of our D7 projects. Visually I have seen {queue} table rows are cleared when the cron runs. So presume it is yes. –  Sivaji Jan 9 '13 at 10:32

4 Answers 4

up vote 7 down vote accepted

When Drupal runs cron tasks, it automatically handles any cron queue defined from modules, in drupal_cron_run(); first the implementations of hook_cron() are invoked, and then the cron queues are emptied.

What you can do is, in your mymodule_cronapi(), adding an entry for another function that handles the cron queue of your module.

function mymodule_cronapi($op, $job = NULL) {
  $items = array();

  $items['queue_users_for_synch'] = array(
    'description' => 'Queue all user accounts for synching.',
    'rule' => '0 3 * * *', // Run this job every day at 3am.
    'callback' => 'mymodule_queue_all_users_for_synching',
  );

  $items['clean_queue'] = array(
    'description' => 'Clean the queue for the user synching.',
    'rule' => '0 4 * * *', // Run this job every day at 4 AM.
    'callback' => 'mymodule_clean_queue',
  );

  return $items;
}

function mymodule_clean_queue() {
  $queues = module_invoke('mymodule', 'cron_queue_info');
  drupal_alter('cron_queue_info', $queues);

  // Make sure every queue exists. There is no harm in trying to recreate an
  // existing queue.
  foreach ($queues as $queue_name => $info) {
    DrupalQueue::get($queue_name)->createQueue();
  }

  foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }
}

The alternative is letting Drupal handle the cron queue for you, but that happens when Drupal cron tasks are executed. If you want to empty the cron queue of your module more frequently, you can only add a new cron task handled by the Elysia Cron module.

The Elysia Cron module handles the cron queues in elysia_cron_run(); this function is being invoked from elysia_cron_cron() (an implementation of hook_cron()), drush_elysia_cron_run_wrapper() (a Drush command callback), and from its own cron.php. If you followed the instructions in the INSTALL.txt file (in particular in "STEP B: CHANGE SYSTEM CRONTAB (OPTIONAL)"), and replaced any invocation of http://example.com/cron.php with http://example.com/sites/all/modules/elysia_cron/cron.php, the Elysia Cron module should be already handling the cron queues. The code I suggested could be used to speed up handling the cron queues used from your module, if there is effectively the need.

// This code is part of the code executed from modules/elysia_cron/cron.php.
define('DRUPAL_ROOT', getcwd());

include_once DRUPAL_ROOT . '/includes/bootstrap.inc';
drupal_override_server_variables(array(
  'SCRIPT_NAME' => '/cron.php',
));
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

if (!isset($_GET['cron_key']) || variable_get('cron_key', 'drupal') != $_GET['cron_key']) {
  watchdog('cron', 'Cron could not run because an invalid key was used.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
elseif (variable_get('maintenance_mode', 0)) {
  watchdog('cron', 'Cron could not run because the site is in maintenance mode.', array(), WATCHDOG_NOTICE);
  drupal_access_denied();
}
else {
  if (function_exists('elysia_cron_run')) {
    elysia_cron_run();
  }
  else {
    drupal_cron_run();
  }
}
share|improve this answer
    
Ah, thanks @kiam! That's what I suspected, but I couldn't quite wrap my brain around it. –  joe_flash Jan 6 '13 at 2:13
    
Actually, I think I'm missing something here. You said the alternative is letting Drupal handle the cron queue for me; I guess part of my original question is when does that actually happen? Each time crontab requests cron.php? If that's the case, that happens every minute (see my first comment on @David's reply). –  joe_flash Jan 6 '13 at 2:42
1  
It's worth mentioning it appears Elysia cron has its own cron_queue processor implementation in elysia_cron_run with cron queues being automatically processed everytime Elysia's cron.php is requested. –  David Thomas Jan 6 '13 at 5:26
    
@joe_flash I am sorry: I should have been clearer. Yes, Drupal runs cron tasks when cron.php is invoked (for every Drupal version until Drupal 7). In Drupal 8, cron.php is not present anymore, and cron tasks are executed using a different system path. –  kiamlaluno Jan 6 '13 at 9:22

The queue will be populated via the Elysia cronapi hook at the set time.

However, the queue will be processed whenever the standard Drupal cron run occurs.

See this worker callback processing snippet at the end of core: drupal_cron_run

 foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }
share|improve this answer
    
David, perhaps Elysia introduces a bit of complication here? I've set crontab to trigger the Elysia cron.php script every minute, which allows Elysia to control task times with minute-wise resolution. No task actually runs every minute though--which is what made me think I needed to create a task to specifically work on queues? –  joe_flash Jan 6 '13 at 2:05
    
@joe_flash as long as drupal_cron_run is being called, your cron queue worker callback will be processed. –  David Thomas Jan 6 '13 at 2:11
    
Ah, I think you're right. However, drupal_cron_run doesn't get called from the Elysia cron.php script (when Elysia is enabled); elysia_cron_run is used instead. –  joe_flash Jan 6 '13 at 2:25
    
In that case, it looks like you can't use hook_cron_queue_info with Elysia cron, unless you specify your own worker callback, as per core drupal_cron_run function snippet above. –  David Thomas Jan 6 '13 at 2:39
    
elysia_cron_run doesn't call drupal_cron_run, but it does call module_invoke_all('cron_queue_info') and does some fancy-pants multi-channel handling that makes smoke come out my ears. –  joe_flash Jan 6 '13 at 2:47

I don't use Elysia, but my solution have always been something like this:

function mymodule_cron() {
  $queue = DrupalQueue::get('mymoudule_queue');
  $queue->createQueue();
  $item = $queue->claimItem(300);

  if (!empty($item->data)) {

    // Do the work.

    if ($sucess) {
      $queue->deleteItem($item);
      watchdog('mymodule', 'It worked.');
    }
    else {
      watchdog('mymodule', 'It did not work!', array(), WATCHDOG_ALERT);
    }
  }
}

It only handles one item, for each cron run. Maybe you want to change that.

share|improve this answer

I've also been trying to wrap my head around this, as I'm using the Queue API for the first time along with Elysia cron. Upon closer inspection you can see that Elysia cron executes queue items when the function elysia_cron_run is called. See this snippet from line 1044 within the file elysia_cron.module:

if (EC_DRUPAL_VERSION >= 7) {
  // D7 Queue processing
  foreach ($queues as $queue_name => $info) {
    $function = $info['worker callback'];
    $end = time() + (isset($info['time']) ? $info['time'] : 15);
    $queue = DrupalQueue::get($queue_name);
    while (time() < $end && ($item = $queue->claimItem())) {
      $function($item->data);
      $queue->deleteItem($item);
    }
  }
}

This helped demystify queue processing for me when using Elysia cron.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.