Creating a Batch with multiple functions

Question

i'm trying to optimise my setup of a Drupal Batch process.

I am reading in thousands of rows of data from an Excel spreadsheet, from multiple worksheets, and creating nodes from the Rows. My import functions work fine, and I can create a Batch, but at present I am creating a batch operation for each row. This means that my batch is very slow (because I'm having to bootstrap Drupal on every row).

What I can't seem to figure out is a way to setup my batch to deal with 'chunks' rather than individual rows.

An example of my import function (of which there are 18) is:

function governbim_coordinate($value){
    module_load_include('inc','phpexcel');
    global $user;
    $project = $_SESSION['governbim']['projnid'];
//Load the required worksheet from the excel file passed to the function as $value
    $result = phpexcel_import($value,TRUE,TRUE,array('setLoadSheetsOnly' => array('Coordinate')));
//Define some common values for our entity
    $values = array(
            'type' => 'coordinate',
            'uid' => $user->uid,
            'status' => 1,
            'comment' => 0,
            'promote' => 0,
            );
//step through each row in the worksheet
    foreach($result['Coordinate'] as $coordinate){
        $title = $coordinate['Name'];
//check to see if this is a new row or if we are updating an existing entity
        $action = governbim_node_actions_node_check('assembly',$title);
        switch($action['op']){
            case 'new':
                $e = entity_create('node',$values); 
                $entity = entity_metadata_wrapper('node',$e);
                break;
            case 'update':
                 $e = entity_create('node',$values); 
                   $entity = entity_metadata_wrapper('node',$action['nid']);
                break;
        }
//*** Here is where we write everything to the database with the entity values and $entity->save() ****/
    }
}

My batch is setup (currently) as:

function governbim_excel_import_batch(){
    drupal_set_message("Importing Data...");
    $path = 'public://imports/';
    $file = 'COBie-UK-2012-example1.xls';
    $datafile = $path.$file;

       $operations[]=array('governbim_coordinate',array($datafile));

        //another 17 $operations[] defined as above

    $batch = array(
        'operations' => $operations,
        'init_message' => t('Constructing batch operation...'),
        'progress_message' => t('Importing Excel data'),
        'finished' => 'governbim_excel_import_batch_finished',
        'error_message' => t('Importing of Excel data encountered an error.'),
        );
    return $batch;
}

I realise I need to get some $context stuff into the import function, but I can't seem to figure out how to setup that aspect properly.

Is anyone able to assist me in re-tuning things so that I can get the batch (with 18 function calls) to display progress of the overall batch and get the batch to execute (say) groups of iterations (as in $limit = 10)?

Answer 1

I do one batch operation but with multiple entry.

/**
 * Batch start function.
 */
function mymodule_batch_start() {
  $batch = array(
    'title' => t('Importing Excel data'),
    'operations' => array(
      array('_mymodule_import', array()),
    ),
    'progress_message' => t('Importing Excel data. Operation @current out of @total.'),
    'error_message' => t('Error!'),
    'finished' => 'mymodule_batch_finished',
  );
  // Run batch in background
  batch_set($batch);

Here is the pattern fot the operation:

/**
 * Import operation.
 */

function _mymodule_import(&$context) {

  // Get your import data with some custom function.
  $import_data = _mymodule_get_import_data($context);

  // Get Count of items
  if (empty($context['sandbox']['progress'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['max'] = count($import_data);
    $context['results']['total_count'] = $context['sandbox']['max'];
  }
  // Create Iteration variable
  if (empty($context['sandbox']['iteration'])) {
    $context['sandbox']['iteration'] = 0;
  }

  // Check for the end of cycle
  if ($context['sandbox']['iteration'] < $context['sandbox']['max']) {

    // Count of operation in one butch step
    $limit = 10;
    // Counts completed operations in one butch step
    $counter = 0;
    if ($context['sandbox']['progress'] != 0) {
      $context['sandbox']['iteration'] = $context['sandbox']['iteration'] + $limit;
    }

    // Main loop of all import items
    for ($i = $context['sandbox']['iteration']; $i < $context['sandbox']['max'] && $counter < $limit; $i++) {


      // Do smth custom here with import data item $import_data[$i].
      _mymodule_handle_import_item($import_data[$i]);


      // Update Progress
      $context['sandbox']['progress']++;
      $counter++;
      // Messages
      $context['message'] = t('Item %progress of %count', array(
        '%progress' => $context['sandbox']['progress'],
        '%count' => $context['sandbox']['max'],
      ));
      $context['results']['processed'] = $context['sandbox']['progress'];
    }
  }

  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }
}

P.S. Sometines I need to import data and then remove some old data. You can store some results of 1st operation in $context and then use it in other operation. I can do import in one batch with two operations:

'operations' => array(
  array('_mymodule_import', array()),
  array('_mymodule_remove_data_after_import', array()),
),