I need to run Batch API on a huge dataset. Specifically I have a custom table with a lot of rows which I need to iterate over. The batch process works fine using small numbers. When using the full set I run out of memory before it even starts because I am loading a very large array of IDs to pass in to the batch function. Think of a 5 million item array. What is a good way to handle this? Break it into smaller chunks somehow to then pass in to Batch API? Increase the memory limit to an (ungodly) amount?
Add a column for 'processed'. Each time you run the batch process select rows with a limit of 100 unprocessed rows for you batch process and then mark each row as processed.
You could run this on cron or something or run a nested batch.