How to manage by cron a long-running queue of very many items

Question

I'm creating a File Checker module that verifies files managed by Drupal actually exist at the uri where Drupal believes they are.

It places an item in a queue for each file entity. As there could be large amounts of files on a site, so there can be large amounts of queue items (10000, 100000 ...). Each item (file entity) can take a few seconds to process if the files are stored remotely e.g. on Amazon S3 or Dropbox. So the queue could take hours to process.

File checking can be scheduled to happen at regular intervals, i.e. by cron, so queue processing needs to happen by cron queueWorkers.

What is the best way to handle the duration of these cron queueWorkers? Their plugin annotation has a cron time option (default 15 seconds) that controls how long they run for when awoken by cron. I could either:

1) set their time option to something very long like 86400, so they can process queue items for a whole 24 hours if they need to; or

2) set their time short like 60 seconds, and use something like the Ultimate Cron module to specify that just this queueWorker is woken up every 60 seconds (and not other cron tasks).

Answer 1

The problem with using queue for an operation like this is that you have to fill up the queue frequently.

Queue makes sense if you have something that is e.g. user or somehow else triggered one by one (or a few at a time), added to the queue and then processed in background. For something like your use case, you want to process tens of thousands of entries, it is a lot of overhead to add them to the queue first and then process them again.

Instead, a simple approach would be to keep track of the file ID. process the first 100 files, sorted by fid, ascending. store the last file ID in state, then on the next run get the next 100 files, where fid is bigger then the last processed fid. If you reach the end (identified based on having processed less than 100 files, you reset the fid back to 0 (possibly combined with some kind of timestamp, so you only start again after N seconds passed since you started the last time). 100 could be configurable, so users can process fewer or more files per run, depending on whether they have them local or not.

Keep in mind that by default, there is only one cron process, if you're running for a long time, it will block everything else. Ultimate Cron could help there, also with running your cron job more frequently than others. It supports multiple threads, but I honestly don't know how well that works already in 8.x (And I am the 8.x maintainer of that module ;)).

You could also always expose it as a drush command, then it would be easy to do it as a completely separate, standalone cron process.