I've run into this both with 3rd party contrib modules as well as some of my own operations. I'm curious at the various ways to speed up my/contrib batch operations?
Assume they work with nodes (import/update etc) and we're dealing with parsing lists of nodes in the 10,000+ range (although I've had to deal with 15 million rows.. which yes - I'm just screwed on..)
Is it quicker to attach to drupals cron.php job and run "headless"? Using Drush? or is this simply a question of how efficient and quick parsing I can develop my code and there are no outside influences or batch specific optimization tips ...
Currently I've run into operations that (using some rough calculation) could take 24+ hours...
This doesn't work for contrib code, but if it's your code and you know it well, I recommend writing a drush command to do the work. Within drush, limit drupal_bootstrap() to the appropriate bootstrap level. I can't recall the actual numbers, but a very large percentage of time for every drupal request is spent in bootstrap, and you can save a lot of time there.
Furthermore, check out the guts of the Migrate module. I don't know how it does it's mojo (never taken the time to grok it), but it can blaze through huge batches of nodes very quickly.
Every batch call is an HTTP request. So you need to find the perfect blend of how many iterations you can process before another HTTP request gets fired. Two things to consider are memory and max execution time. You'll want to process as many iterations as possible per batch to reduce the number of HTTP requests as they are most likely the culprit to your slow batch.
As others have said Drush is a good solution, but a queue is a great tool to use. Batch API in Drupal 7 uses the built-in core Queue API so if you're using MySQL your process could be bottlenecked there. But, Drupal 7's Queue API is pluggable, so you could use another queue system like beanstalkd.
If you can make it parallel thats a good start. Here are some of my thoughts on this as I've used 4 threads to crawl over a million pages before (via boost). Looking to make it generalized now. http://groups.drupal.org/node/126624