13

I've run into this both with 3rd party contrib modules as well as some of my own operations. I'm curious at the various ways to speed up my/contrib batch operations?

Assume they work with nodes (import/update etc) and we're dealing with parsing lists of nodes in the 10,000+ range (although I've had to deal with 15 million rows.. which yes - I'm just screwed on..)

Is it quicker to attach to drupals cron.php job and run "headless"? Using Drush? or is this simply a question of how efficient and quick parsing I can develop my code and there are no outside influences or batch specific optimization tips ...

Currently I've run into operations that (using some rough calculation) could take 24+ hours...

Thanks!

Improve this question

4 Answers 4

Reset to default
10

This doesn't work for contrib code, but if it's your code and you know it well, I recommend writing a drush command to do the work. Within drush, limit drupal_bootstrap() to the appropriate bootstrap level. I can't recall the actual numbers, but a very large percentage of time for every drupal request is spent in bootstrap, and you can save a lot of time there.

Furthermore, check out the guts of the Migrate module. I don't know how it does it's mojo (never taken the time to grok it), but it can blaze through huge batches of nodes very quickly.

Improve this answer
1
  • Thanks for the input - I'll be looking into the migrate module more and that drupal_boostrap was a great tip as well ;)
    – electblake
    Mar 30, 2011 at 21:08
9

Every batch call is an HTTP request. So you need to find the perfect blend of how many iterations you can process before another HTTP request gets fired. Two things to consider are memory and max execution time. You'll want to process as many iterations as possible per batch to reduce the number of HTTP requests as they are most likely the culprit to your slow batch.

If your batch is just too heavy to run efficiently you could try using a queue instead. There is a good batch vs. queue presentation here http://sf2010.drupal.org/conference/sessions/batch-vs-queue-api-smackdown. Queues do not provide user feedback and can be run in parallel.

If you require user feedback you are tied to batch, but you could even use queue in your batch to try to optimize it.

Improve this answer
0
3

As others have said Drush is a good solution, but a queue is a great tool to use. Batch API in Drupal 7 uses the built-in core Queue API so if you're using MySQL your process could be bottlenecked there. But, Drupal 7's Queue API is pluggable, so you could use another queue system like beanstalkd.

Improve this answer
2

If you can make it parallel thats a good start. Here are some of my thoughts on this as I've used 4 threads to crawl over a million pages before (via boost). Looking to make it generalized now. http://groups.drupal.org/node/126624

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.