I need to run Batch API on a huge dataset. Specifically I have a custom table with a lot of rows which I need to iterate over. The batch process works fine using small numbers. When using the full set I run out of memory before it even starts because I am loading a very large array of IDs to pass in to the batch function. Think of a 5 million item array. What is a good way to handle this? Break it into smaller chunks somehow to then pass in to Batch API? Increase the memory limit to an (ungodly) amount?
5
-
Five million numeric IDs shoudln't really be enough to collapse your system. Are you sure the array is containing only IDs? If it is, then you could try increasing your memory, or if that's not possible, query your IDs in batches themselves.– JaypanJan 11, 2019 at 4:17
-
2The batch should be batched, don’t try to do millions in one batch callback. Increasing the memory limit beyond 256MB will not have much effect.– KevinJan 11, 2019 at 4:22
-
3You may also want to consider the Queue API rather than the Batch API. This will be a little more reliable, and not require browser interaction.– JaypanJan 11, 2019 at 4:26
-
1I've processed millions of records using Batch API, the trick is only loading it a couple of hundred records (or whatever's appropriate for your record size) at a time.– AdaddinsaneJan 11, 2019 at 8:59
-
One "gotcha" I've run into with Batch API is that even a single Warning message being logged will cause the memory to run out. I would double check your Watchdog logs and make sure you're getting zero notices, warnings etc., because even 1 notice per record processed can put you into memory hell!– Tyler FaheyJan 11, 2019 at 14:44
Add a comment
|
1 Answer
Add a column for 'processed'. Each time you run the batch process select rows with a limit of 100 unprocessed rows for you batch process and then mark each row as processed.
You could run this on cron or something or run a nested batch.
