4

Is there a way to use Drupal Queue API to sequence the execution of tasks, like in a pipeline?

My use case is pretty simple: I have a number of tasks executing in the background, doesn't matter their order because they are self-contained. However, I'd like a single task (of a different function) to execute after all the others are done.

How would this be done?

1

(Note: I've worked with queue in d6 but not in d7 yet; I don't know if there's a difference.)

My case was a little different, but I did need to impose a sequence on some items. In my schema, I have a field for sequencing (actually a next_attempt time). In createItem(), when adding an item to the queue, I set that field. Then in claimItem(), my query to fetch an item to work on checks that field as well as expire.

  • Thanks. So if I understand correctly, you're saying: make my own implementation of DrupalQueueInterface that does what I want. – infojunkie Apr 8 '13 at 15:46
  • Yes, that's my approach for cases where you need to have control during the processing of the queue. – RSTaylor Apr 8 '13 at 16:19
  • Thanks - this makes sense to me. I'll use this approach. – infojunkie Apr 8 '13 at 16:22
2

Together with a bunch of other people we created the queue runner and it does support this use-case and we use it internally. You can add a collector task that includes all your small tasks and you add a second collector task so it is executed after the whole lot of small tasks. See the API file for the queue runner to figure out how you should do this exactly. It works with the default Drupal queue class so you can even extend it with external queue systems such as http://drupal.org/project/aws_sqs (amazon queue class) or others.

http://drupal.org/project/queue_runner Works for D6 and D7

1

According to the documentation, the best you can get from the Queue operations API is ordered tasks and at least one execution of each task. However, the documentation does not specify how ordering apply. A reliable queue that guarantee the execution of an item only starts after the execution of all previous items has started would comply with this specification but would not be suitable for your use case.

You could solve this with some sort of tasks counter. When en-queuing the self-contained tasks, you increase their shared counter by the number of en-queued tasks. At the end of each task, you decrease that counter. If the counter reach zero, you can safely en-queue the final task, knowing that all previous self-contained task has been completed.

Finally, the documentation states that even a reliable queue will execute each item at least once. So the same self-contained task could be executed twice. Since you want all your self-contained tasks to complete before execution of the final task, you will need to add a safeguard to avoid the same self-contained task to decrease the counter twice.

  • Thanks. I will try to implement my own DrupalReliableQueueInterface that uses the static counter technique, or some other sequencing mechanism. – infojunkie Apr 8 '13 at 16:16
0

To execute a single task after all others, use a different queue with that single task wrapped in a worker which peeks at the other queue for emptiness and runs the single task when empty, otherwise does nothing. Fine tune in hook_cron_queue_info how often you peek.

  • Thanks. How would that work if I fire two pipelines simultaneously? There would be two end tasks waiting to fire. – infojunkie Apr 14 '13 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.