0

i'm trying to optimise my setup of a Drupal Batch process.

I am reading in thousands of rows of data from an Excel spreadsheet, from multiple worksheets, and creating nodes from the Rows. My import functions work fine, and I can create a Batch, but at present I am creating a batch operation for each row. This means that my batch is very slow (because I'm having to bootstrap Drupal on every row).

What I can't seem to figure out is a way to setup my batch to deal with 'chunks' rather than individual rows.

An example of my import function (of which there are 18) is:

function governbim_coordinate($value){
    module_load_include('inc','phpexcel');
    global $user;
    $project = $_SESSION['governbim']['projnid'];
//Load the required worksheet from the excel file passed to the function as $value
    $result = phpexcel_import($value,TRUE,TRUE,array('setLoadSheetsOnly' => array('Coordinate')));
//Define some common values for our entity
    $values = array(
            'type' => 'coordinate',
            'uid' => $user->uid,
            'status' => 1,
            'comment' => 0,
            'promote' => 0,
            );
//step through each row in the worksheet
    foreach($result['Coordinate'] as $coordinate){
        $title = $coordinate['Name'];
//check to see if this is a new row or if we are updating an existing entity
        $action = governbim_node_actions_node_check('assembly',$title);
        switch($action['op']){
            case 'new':
                $e = entity_create('node',$values); 
                $entity = entity_metadata_wrapper('node',$e);
                break;
            case 'update':
                 $e = entity_create('node',$values); 
                   $entity = entity_metadata_wrapper('node',$action['nid']);
                break;
        }
//*** Here is where we write everything to the database with the entity values and $entity->save() ****/
    }
}

My batch is setup (currently) as:

function governbim_excel_import_batch(){
    drupal_set_message("Importing Data...");
    $path = 'public://imports/';
    $file = 'COBie-UK-2012-example1.xls';
    $datafile = $path.$file;

       $operations[]=array('governbim_coordinate',array($datafile));

        //another 17 $operations[] defined as above

    $batch = array(
        'operations' => $operations,
        'init_message' => t('Constructing batch operation...'),
        'progress_message' => t('Importing Excel data'),
        'finished' => 'governbim_excel_import_batch_finished',
        'error_message' => t('Importing of Excel data encountered an error.'),
        );
    return $batch;
}

I realise I need to get some $context stuff into the import function, but I can't seem to figure out how to setup that aspect properly.

Is anyone able to assist me in re-tuning things so that I can get the batch (with 18 function calls) to display progress of the overall batch and get the batch to execute (say) groups of iterations (as in $limit = 10)?

0

I do one batch operation but with multiple entry.

/**
 * Batch start function.
 */
function mymodule_batch_start() {
  $batch = array(
    'title' => t('Importing Excel data'),
    'operations' => array(
      array('_mymodule_import', array()),
    ),
    'progress_message' => t('Importing Excel data. Operation @current out of @total.'),
    'error_message' => t('Error!'),
    'finished' => 'mymodule_batch_finished',
  );
  // Run batch in background
  batch_set($batch);

Here is the pattern fot the operation:

/**
 * Import operation.
 */

function _mymodule_import(&$context) {

  // Get your import data with some custom function.
  $import_data = _mymodule_get_import_data($context);

  // Get Count of items
  if (empty($context['sandbox']['progress'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['max'] = count($import_data);
    $context['results']['total_count'] = $context['sandbox']['max'];
  }
  // Create Iteration variable
  if (empty($context['sandbox']['iteration'])) {
    $context['sandbox']['iteration'] = 0;
  }

  // Check for the end of cycle
  if ($context['sandbox']['iteration'] < $context['sandbox']['max']) {

    // Count of operation in one butch step
    $limit = 10;
    // Counts completed operations in one butch step
    $counter = 0;
    if ($context['sandbox']['progress'] != 0) {
      $context['sandbox']['iteration'] = $context['sandbox']['iteration'] + $limit;
    }

    // Main loop of all import items
    for ($i = $context['sandbox']['iteration']; $i < $context['sandbox']['max'] && $counter < $limit; $i++) {


      // Do smth custom here with import data item $import_data[$i].
      _mymodule_handle_import_item($import_data[$i]);


      // Update Progress
      $context['sandbox']['progress']++;
      $counter++;
      // Messages
      $context['message'] = t('Item %progress of %count', array(
        '%progress' => $context['sandbox']['progress'],
        '%count' => $context['sandbox']['max'],
      ));
      $context['results']['processed'] = $context['sandbox']['progress'];
    }
  }

  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }
}

P.S. Sometines I need to import data and then remove some old data. You can store some results of 1st operation in $context and then use it in other operation. I can do import in one batch with two operations:

'operations' => array(
  array('_mymodule_import', array()),
  array('_mymodule_remove_data_after_import', array()),
),

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.