2

Can the batch API continue to run once the browser closes? If not then can I run a batch with Drush? I have an insane amount of data to migrate and I don't really have time to wait for it to import every item one at a time on my local machine.

Currently I have my batch process setup to import each item (groups, users, blogs, files, etc) one per op call. But I am going to change that to do 50 to 100 per call to improve the speed. But I just ran it last night on just my groups and it took almost an hour to run on almost 800 groups.

This is part of a data migration from multiple websites to one unified site. Thoughts? Suggestions? Can Drush help me out?

EDIT: I have rolled my own export/import scripts for this because of the nature of the migration (multiple Drupal 4.7 sites that are highly customized and with potentially hacked cores to Drupal 6). I already have the exported data file created and ready to go. I am just testing the import portion of this madness.

Improve this question
2
  • You don't say what kind of migration you're doing. If you're using either feeds or migrate, drush commands are built in. If you've rolled your own, convert it to something that uses drupal queue, as bojanz says below.
    – rfay
    Aug 4, 2011 at 1:56
  • I have rolled my own. I already have the data stored in files from the source sites. Basically what I did was create an output file for each data type that I am pulling that consisted of a serialized PHP array of values with the filename identifying the data type and source site.
    – Patrick
    Aug 4, 2011 at 18:35

3 Answers 3

Reset to default
3

It seems possible, have you tried: Processing batch jobs with drush.

Improve this answer
2
  • I have tried that and it seems to be acting a little better. That said I think I am also having trouble with a contrib. PHP memory usage is ballooning like crazy when I run it in Drush and the number never goes down.
    – Patrick
    Aug 4, 2011 at 18:32
  • Also that link above is what I have based my Drush implementation on. Though since I am building it into what I have already developed for the batch API I was able to greatly simplify my code for Drush.
    – Patrick
    Aug 4, 2011 at 18:37
3

Sounds like a job for Drupal Queue then. Explore that as an alternative to Batch API. It is included in Drupal 7, and there is a Drupal 6 backport in contrib.

Improve this answer
1
  • I don't understand the point of the Queue module when it seems to do the same thing as the batch API. What is the point of it?
    – Patrick
    Aug 4, 2011 at 18:35
2

I've successfully used drush with the Batch API in exactly the situation you describe, importing content from multiple sites into one site. The total content imported included 800-some user accounts, a few thousand nodes, etc. The total execution time was something like 30 minutes.

I essentially wrote a number of php scripts, one per content type (or other thing), to load the nodes (or other things) from the source sites' databases and save them using the API. I used the pattern in Processing batch jobs with drush for these scripts, and then used drush commands in a shell script to run them all.

Improve this answer
2
  • 1
    Oh, I forgot to mention--the biggest time sink in these scripts was using API like node_load() or user_load(). If at all possible, you might want to fetch everything of a type in a single, custom query instead of using the API on the source sites. (Still save using the API, of course.)
    – xjm
    Aug 4, 2011 at 17:09
  • Yeah all of my data has already been exported. I just edited my post above to include information about it.
    – Patrick
    Aug 4, 2011 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.