2

I need to run Batch API on a huge dataset. Specifically I have a custom table with a lot of rows which I need to iterate over. The batch process works fine using small numbers. When using the full set I run out of memory before it even starts because I am loading a very large array of IDs to pass in to the batch function. Think of a 5 million item array. What is a good way to handle this? Break it into smaller chunks somehow to then pass in to Batch API? Increase the memory limit to an (ungodly) amount?

  • Five million numeric IDs shoudln't really be enough to collapse your system. Are you sure the array is containing only IDs? If it is, then you could try increasing your memory, or if that's not possible, query your IDs in batches themselves. – Jaypan Jan 11 at 4:17
  • 2
    The batch should be batched, don’t try to do millions in one batch callback. Increasing the memory limit beyond 256MB will not have much effect. – Kevin Jan 11 at 4:22
  • 3
    You may also want to consider the Queue API rather than the Batch API. This will be a little more reliable, and not require browser interaction. – Jaypan Jan 11 at 4:26
  • 1
    I've processed millions of records using Batch API, the trick is only loading it a couple of hundred records (or whatever's appropriate for your record size) at a time. – Adaddinsane Jan 11 at 8:59
  • One "gotcha" I've run into with Batch API is that even a single Warning message being logged will cause the memory to run out. I would double check your Watchdog logs and make sure you're getting zero notices, warnings etc., because even 1 notice per record processed can put you into memory hell! – Tyler Fahey Jan 11 at 14:44
1

Add a column for 'processed'. Each time you run the batch process select rows with a limit of 100 unprocessed rows for you batch process and then mark each row as processed.

You could run this on cron or something or run a nested batch.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.