1

I use ultimate cron to fill in a queue with some nids.

To add the items in the queue, I use something like:

$queue = DrupalQueue::get('my_queue');
foreach($urls as $nid=>$url){
  sleep(1);
  $response = drupal_http_request($url,array('timeout'=>3,'method'=>'head'));
  if($response->code=='404'){
    $queue->createItem(array('nid' => $nid));
  }

This works OK.

When the time comes, another cron job is executed that deletes those nodes. This is done like:

$queue = DrupalQueue::get('my_queue');
for($i=0; $i<$queue->numberOfItems(); $i++){
  $item = $queue->claimitem();
  node_delete($item->data['nid']);
  $queue->deleteItem($item);
}

This is not working OK; there is a small issue: Everytime the second job runs, only half of the items in the queue are processed (deleted). If there are 3000 items in the queue, only 1500 are deleted. If there are 1500 items, only 750 are deleted and so on.

Why is that? Is there a way to control it?

  • Did you try increasing the timeout? – heshanlk Jul 2 '13 at 20:44
  • @heshanlk This can not be the timeout. It happens even with 3 items in the queue. – Mike Jul 2 '13 at 20:55
  • I think it could be timeout related as your trying to fetch a remote URL with a 3 second timeout ... drupal_http_request() will block for all output to be sent to a buffer. – tenken Jul 2 '13 at 21:00
  • This is for inserting items into the queue, which works ok. The problem is when removing items from the queue. – Mike Jul 2 '13 at 21:20
2

So you have 2 queues walking all over eachother. The timeout value is how long the item is Leased to have work done upon it. The lease-time does not guarantee your processing will finish it time. A Leased item will not be able to be deleted until its lease has expired.

So you should analyze the tasks being run and adjust your timeout to be larger than the longest processing time you have recorded as occurring. I typically do a 2x or 10x as much Lease-time as my longest evaluated processing time. Once an item has completed processing its Lease is expired (so over guessing doesn't really hurt you here).

Alternatively, instead of having 2 queues you could do all this with 1 queue. To try to get around contention. You could insert into a single queue all your NIDs with an additional value of task; and at the end of that queue append a new list of NIDs with an additional flag value of delete or purge. Adjust your worker function to examine a NID and the action to perform on the NID; task or delete.

For example your createItem could look like:

// Add all NIDs for task to be performed.
$queue->createItem(array('nid' => $nid, 'action' => 'do-task'));
// in the same list later on queue the same NIDs to be deleted.
// by setting a huuuuge lease time you will never accidently purge an
// uncompleted task.
$queue->createItem(array('nid' => $nid, 'action' => 'do-delete'));

You may find working with a single queue -- easier than juggling 2 queues.

|improve this answer|||||
  • I use a single queue named: "my_queue" not two. And I do not want to perform different tasks, just delete the nodes. Also, the default lease-time of 1 hour never prevents me from removing the item from the queue. – Mike Jul 2 '13 at 21:17
1

I encountered this same issue. I fixed it by setting $queue->numberOfItems() into it's own variable and then using the variable in the foreach. If I didn't do this, I found that the number of items was decrementing for each 'for' iteration, so only half were being processed.

So change:

for($i=0; $i<$queue->numberOfItems(); $i++){
  $item = $queue->claimitem();
  node_delete($item->data['nid']);
  $queue->deleteItem($item);
}

to:

$numberOfQueueItems = $queue->numberOfItems();
for($i=0; $i<$numberOfQueueItems; $i++){
  $item = $queue->claimitem();
  node_delete($item->data['nid']);
  $queue->deleteItem($item);
}

I know I'm 4 years too late, but hope this helps the next person.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.