3

This question already has an answer here:

I am developing a Drupal Cron that has a very long-running task. First I tried with a simple Cron, which, of course, did not work because of "Memory exhausted" PHP fatal error.

Then, I developed a Batch with the Batch API which worked fine as a normal Batch. However, I could not make to run it completely inside the Cron. It receives the same PHP error.

After that, I developed a Cron using the Queue API. However, I could not make Drupal to run all queue jobs in the same cron. The execution stops at some point and I am unable to run more jobs. I know that there is an option to define the worker's function time, but my task is a long-running task. So it is limited by PHP.

I also tried to use Background Process module in order to execute my long-running task but it had the same problem as before: the execution stops at some point.

I am sure that the code can be executed completely. The only working way was through Batch page but I want to run it automatically every day (Cron).

Do you know how can I run a Cron until it finishes?

As I said, I tried with Batches and Queues but it did not work. PHP "memory exhausted" or timeout errors appear or the execution can be cut by Drupal.

Thank you.

marked as duplicate by kenorb, Adrian Cid Almaguer, Shawn Conn, Krishna Mohan, Scott Joudry Mar 21 '16 at 16:36

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • If you're batching your tasks and still receiving an out-of-memory error, you need to do one (or more) of these 3 things: 1. Reduce the number of items per batch run 2. Improve the performance of the code in each single operation 3. Increase the amount of physical/virtual memory on your server – Clive Feb 2 '16 at 16:15
  • Hi Clive, if I execute the long-running task as a batch, it works perfectly. The problem comes when I want to execute that long-running task as a cron. If I simulate the Batch Form Submit inside the Cron, at some point the execution ends for no reason. If I execute the Batch in its own normal page, it works. – MmMnMr Feb 2 '16 at 16:20
  • Hi kenorb, the "Memory exhausted" error came from the execution of the long-running task and PHP memory limitation. This problem was no related with a specific module issue. The question you linked is related with several WSOD that may be produced by a specific module. In any case, thank you for your response. – MmMnMr Mar 12 '16 at 19:36
  • 1
    Please, this question IS NOT a duplicate from "White screen of death: Fatal error: Allowed memory size of X bytes exhausted". Please, read my question and answer. This issue is different. – MmMnMr Sep 4 '16 at 14:10
3

After a long battle analyzing how to execute a long-running task as a Drupal Cron, finally I found a generic solution that involves the Queue API and Cron. Maybe this can help other people who is struggling with a similar problem. Here are the steps I followed:

  1. Think about how to execute your long-running task as chunks in terms of Drupal Queue items. You need to represent all tasks as queue items. For example, the first item (or action) may be to initialize the complex work that needs to be done. Then, the following items may execute that work by pieces. Each item (or chunk) has to be processed individually, so no PHP Timeout or PHP Memory Exhausted error appear.

  2. Create a new Cron that will create the queue and its items. Make sure that you also implement the hook_cron_queue_info().

  3. Implement the queue item process function you defined in the above hook. If you need to pass information between each item process, you can use Drupal's variable_get(...) and variable_set(...). If you need a more complex structure, you can define a database with Schema API in the module my_module.install file.

  4. Now create a new function that will read and process one or more items of your queue (if not empty).

For example:

function my_module_process_my_queue() {
  $my_queue = DrupalQueue::get('my_module_queue');
  if ($my_queue->numberOfItems() > 0) {
    $action = $my_queue->claimItem();
    if ($action) {
      my_module_queue_process_item($action->data);
      $my_queue->deleteItem($action);
    }
  }
}
  1. Install a Cron module (such as Elysia Cron) or add a cron job in your server configuration. At least, you need to specify the execution of 2 Cron functions. The first one (from step 2) will run the moment you prefer. In my case, it has to run once a day. The second one will be a simple Cron that will call the function from step 4. That second Cron will run every X minutes. Where X means the maximum time execution that a queue item processing may take. If you do not install a Cron module, you do not need the second Cron function. In my case, I decided to use Linux crontab configuration and Drush.

For example:

*/2 * * * * cd /path/to/your/drupal/installation && export DRUSH_PHP=/path/to/your/bin/php && drush ev "my_module_process_my_queue();"
  1. Finally, execute your crons or wait until they are executed. Once a day, the queue will be filled with new queue items. Those items will be processed during the first Cron execution and every X minutes with the other one. So, after some time, all queue items will be processed and the long-running task will finish.
  • Very very helpful – Jonathan Aug 31 '16 at 15:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.