I would suggest creating a drush command that retrieves and saves the data via batch or queue. This article may come handy which provides the following code examples:
function mymodule_setup_batch($start=1, $stop=100000) {
// ...
// Populate $lots_of_data from record $start to record $stop.
// ...
//Break up all of our data so each process does not time out.
$chunks = array_chunk($lots_of_data, 20);
$operations = array();
$count_chunks = count($chunks);
//for every chunk, assign some method to run on that chunk of data
foreach ($chunks as $chunk) {
$i++;
$operations[] = array("mymodule_method_to_work_on_a_small_part", array( $chunk ,'details'=> t('(Importing chunk @chunk of @count)', array('@chunk '=>$i, '@count'=>$count_chunks))));
$operations[] = array("mymodule_another_method",array($chunk));
}
//put all that information into our batch array
$batch = array(
'operations' => $operations,
'title' => t('Import batch'),
'init_message' => t('Initializing'),
'error_message' => t('An error occurred'),
'finished' => 'mymodule_finished_method'
);
//Get the batch process all ready!
batch_set($batch);
$batch =& batch_get();
//Because we are doing this on the back-end, we set progressive to false.
$batch['progressive'] = FALSE;
//Start processing the batch operations.
drush_backend_batch_process();
}
function mymodule_method_to_work_on_a_small_part ($chunk, $operation_details, &$context) {
//Do something to $chunk, maybe create a node?
$context['message'] = $operation_details; //Will show what chunk we're on.
}
function mymodule_another_method($chunk, &$context) {
//Do some more work.
$context['message'] = t('We have done a second thing to a chunk of data');
}
function mymodule_finished_method($success, $results, $operations) {
//Let the user know we have finished!
print t('Finished importing!');
}
If you like, you can implement Drush command, e.g.
function mymodule_drush_command() {
$items = array();
$items['myimport'] = array(
'callback' => 'mymodule_setup_batch',
'description' => dt('Import'),
'arguments' => array(
'start' => "start",
'stop' => "stop",
),
);
return $items;
}
function mymodule_drush_help($section) {
switch ($section) {
case 'drush:myimport':
return dt("import items from the Internal Database [start record] [end record].");
}
}
