Long-running cron task never fully executed [duplicate]

Question

I am developing a Drupal Cron that has a very long-running task. First I tried with a simple Cron, which, of course, did not work because of "Memory exhausted" PHP fatal error.

Then, I developed a Batch with the Batch API which worked fine as a normal Batch. However, I could not make to run it completely inside the Cron. It receives the same PHP error.

After that, I developed a Cron using the Queue API. However, I could not make Drupal to run all queue jobs in the same cron. The execution stops at some point and I am unable to run more jobs. I know that there is an option to define the worker's function time, but my task is a long-running task. So it is limited by PHP.

I also tried to use Background Process module in order to execute my long-running task but it had the same problem as before: the execution stops at some point.

I am sure that the code can be executed completely. The only working way was through Batch page but I want to run it automatically every day (Cron).

Do you know how can I run a Cron until it finishes?

As I said, I tried with Batches and Queues but it did not work. PHP "memory exhausted" or timeout errors appear or the execution can be cut by Drupal.

Thank you.

Answer 1

After a long battle analyzing how to execute a long-running task as a Drupal Cron, finally I found a generic solution that involves the Queue API and Cron. Maybe this can help other people who is struggling with a similar problem. Here are the steps I followed:

1. Think about how to execute your long-running task as chunks in terms of Drupal Queue items. You need to represent all tasks as queue items. For example, the first item (or action) may be to initialize the complex work that needs to be done. Then, the following items may execute that work by pieces. Each item (or chunk) has to be processed individually, so no PHP Timeout or PHP Memory Exhausted error appear.

2. Create a new Cron that will create the queue and its items. Make sure that you also implement the hook_cron_queue_info().

3. Implement the queue item process function you defined in the above hook. If you need to pass information between each item process, you can use Drupal's variable_get(...) and variable_set(...). If you need a more complex structure, you can define a database with Schema API in the module my_module.install file.

4. Now create a new function that will read and process one or more items of your queue (if not empty).

For example:

function my_module_process_my_queue() {
  $my_queue = DrupalQueue::get('my_module_queue');
  if ($my_queue->numberOfItems() > 0) {
    $action = $my_queue->claimItem();
    if ($action) {
      my_module_queue_process_item($action->data);
      $my_queue->deleteItem($action);
    }
  }
}

5. Install a Cron module (such as Elysia Cron) or add a cron job in your server configuration. At least, you need to specify the execution of 2 Cron functions. The first one (from step 2) will run the moment you prefer. In my case, it has to run once a day. The second one will be a simple Cron that will call the function from step 4. That second Cron will run every X minutes. Where X means the maximum time execution that a queue item processing may take. If you do not install a Cron module, you do not need the second Cron function. In my case, I decided to use Linux crontab configuration and Drush.

For example:

*/2 * * * * cd /path/to/your/drupal/installation && export DRUSH_PHP=/path/to/your/bin/php && drush ev "my_module_process_my_queue();"

6. Finally, execute your crons or wait until they are executed. Once a day, the queue will be filled with new queue items. Those items will be processed during the first Cron execution and every X minutes with the other one. So, after some time, all queue items will be processed and the long-running task will finish.