1

My D7 module uses hook_cron to mirror a big external datasource locally. For each new or updated row, $queue->createItem() schedules a task requiring several external API round trips to build the locally mirrored row.

I might queue 750 tasks but Drupal only runs 50 on a cron. How do I make all queued tasks run on each cron?

Please note:

  • There are no errors reported in /var/log/apache2/error.log

  • The latest message in admin/reports/dblog reports "Cron run completed." (looks successful)

4
  • It's not obvious to me what role Drupal (modules) play in your setup, or where the 50 limit comes from in the first place.
    – Letharion
    Mar 12, 2012 at 18:28
  • It's a Drupal 7 question in that I'm implementing hook_cron and hook_cron_queue_info to queue these tasks. Mar 12, 2012 at 18:35
  • Now I see. I made a small edit to the question to make it more clear.
    – Letharion
    Mar 12, 2012 at 18:47
  • Thanks @Letharion. I made the hook_cron question more clear now. Mar 12, 2012 at 18:56

2 Answers 2

2

Basically PHP has memory and execution time limitations, so if you try to do a huge import of an external database you'll run out of runtime if not memory.

So the solution is to spread it out over many http requests, each doing a smaller chunk of the job.

For this, Drupal has the Batch API, which you can call from cron as needed.

Here's the Examples project's Batch example (with tests!) to guide the way. And a decent explanation with some code samples.

5
  • 2
    Note that you can still use your existing queue, you just write a wrapper batch API with an operation that processes an item of your queue. You can place a button for that on a settings page or so, if you have one already or create one if not.
    – Berdir
    Mar 12, 2012 at 19:35
  • 1
    Defer to the wisdom of Berdir. :-)
    – paul-m
    Mar 12, 2012 at 19:36
  • Actually, there is a module for that (tm): drupal.org/project/queue_ui
    – Berdir
    Mar 12, 2012 at 19:40
  • Manually running the job would defeat the purpose of using cron. I am reading that the Batch API is not intended for use without the UI. Is that not true? I agree that this large number of heavy tasks should cause PHP to run out of memory or execution time, but I don't have anything reported in /var/log/apache2/eror.log and the only message in admin/reports/dblog is "Cron run completed." Mar 12, 2012 at 20:26
  • When a user initiates a batch process, it shows a nifty progress bar and the user can cancel. You can also use the Batch API with 'background' processes, as a way to launch a chain of http requests for the different tasks.
    – paul-m
    Mar 13, 2012 at 1:10
1

For completeness, two additional solutions:

  • You can define a custom time limit for your queue in hook_queue_info(), that is however limited by PHP's total execution time.

  • You can manually call cron as often as it takes. This is easy for task that you really only have to do once to work through the initial amount of tasks/items. If you have drush, you can execute something like following, which will execute cron over and over again until you manually stop it: while [ 1 == 1 ]; do drush cron; done. There might be nicer ways to do that but it works and it also prevents issues with time execution and memory limits.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.