How do I process a huge list of files?

Question

I have a huge list (> 100 000) of files that need to be processed. I want processing to run in the background and continuously (when it has processed the last, it just restarts on the first).

I've tried to do it with hook_cron(). There is a limit on how long cron can run, so to prevent cron timeouts, I throttle cron to process about 1000 files each time it runs (that is about the max cron can handle without timeouts).

To keep track of things between cron runs I use two db variables:

$mymodule_huge_array_data = variable_get('mymodule_huge_array_data', array());
$mymodule_counter_index = variable_get('mymodule_huge_array_index', 0);

After setting things up, cron get these two variables, and use $mymodule_counter_index to know where to start in the list of files. After processing 1000 files, it does:

$mymodule_counter_index += 1000;
variable_set('mymodule_huge_array_index', $mymodule_counter_index);

My problem is that getting that huge blob of an array from the database at the start of each cron makes the site slow during cron runs - even if I throttle cron to process only a single file.

I've looked at various alternatives. The core batch API (Batch operations) alone can not be run from cron. Job Scheduler just uses cron to schedule jobs, so I don't think it will improve anything here.

Should I replace the single db variable holding the huge array blob with a table (with one file per record)?

I am currently looking Background Process and its Background Batch submodule. Is it suitable for this use case? It doesn't look so, it runs in the background, but just like the core batch API (Batch operations) alone, you need to initiate the process manually.

The accepted answer to this also looks like an option: Do I need a cron task for processing a queue?

Are there other (better) ways to do this?

What is the right way to chop up a large job into smaller bits, keeping track of the state as it progress through operations?

(To prevent this from being closed as "opinion based", let me emphasis that I am not asking for opinions about this. I am asking for the correct way to process a huge list of files with Drupal.)

Answer 1

You can try

Method I - Using Cron and queue

• Cron will run the query and push the list of nodes to the queue
• queue will pick the items one by one and Unpublish them

Method - II Using Cron and batch

• Cron will run the query and push the list of nodes to the Batch process
• Batch process will be launched programmatically using background_process module

References

1. Batch API
2. How to create a batch operation in Drupal 7
3. SE Answer
4. D.ORG post