Batch API timeout during initialization

Question

I'm creating a module to batch import/update nodes in Drupal 7 based on input files that can be XML or other formats. Worst case scenario is they are like ~10-20MB and can contain 10-20k entries.

Currently, the module works fine for small files (~2k entries), but I get timeouts during initialization (before the actual batch job) for bigger files. The input file parsing is not the problem, it works fine and this is an example table of parsing times:

ENTRIES  PARSING (s)  FORMAT
     63        0.056  xml
    544        0.144  xml
   1706        0.202  xml
   2741        0.940  xml  <- batch job doesn't start
   3398        0.419  txt  <- batch job doesn't start
   7916        8.726  xml  <- batch job doesn't start
   8607       29.734  xml  <- batch job doesn't start

This is the relevant part of my module's code:

function batch_import($ps_platform, $ps_input_file) {
    # I've tried to change timeout limit with no luck        
    set_time_limit(500);

    # Parsing the input file to my own class. The time spent for
    # different size files can be in the table above.
    $o_dat = new DatContainer($ps_input_file);

    # [1/2] Preparing the batch job array we will output
    #---------------------------------------------------
    $a_batch_job = array(
        'title' => t('Importing dat file'),
        'init_message' => t('Beginning dat import process'),
        'operations' => array(),
        'finished' => 'batch_finished',
        );

    # [2/2] Populate the operations sub-array with all the jobs we want to do
    # -----------------------------------------------------------------------
    foreach ($o_dat as $o_entry) {
        $a_batch_job['operations'][] = array('process_entry', array($ps_platform, $o_entry));
    }

    batch_set($a_batch_job);
}

The actual function to process each individual task doesn't do anything by now, it simply prints the progress.

function process_entry($ps_platform, $po_entry, &$context) {
    $context['message'] = "Processing: {$po_entry->s_name}";
    $o_node = _get_entry_node_by_id($ps_platform, $po_entry->s_id);
    $_SESSION['http_request_count']++;
}

Does anybody see where is the problem? After checking many examples for batch api, I'm quite frustrated because my code seems to be correct.

Regards and thanks for your time.

Answer 1

The timeout happens because of large amount of operations within $a_batch_job['operations'] array. For every operation defined queue item is created and stored in the queue table of the database.

Check the _batch_populate_queue() method.

Workaround

Instead of defining Pre-set list of calls, you can take a route of Alterable list of calls. So in your case it could be like this:

function batch_import($ps_platform, $ps_input_file) {
  # Parsing the input file to my own class. The time spent for
  # different size files can be in the table above.
  $o_dat = new DatContainer($ps_input_file);

  # [1/2] Preparing the batch job array we will output
  #---------------------------------------------------
  $a_batch_job = array(
    'title' => t('Importing dat file'),
    'init_message' => t('Beginning dat import process'),
    // Pass our parsed file as a parameter to the batch operation callback.
    'operations' => array('process_entry', array($o_dat, $ps_platform)),
    'finished' => 'batch_finished',
  );

  batch_set($a_batch_job);
}

And your process_entry() would look like this:

function process_entry($o_dat, $ps_platform, &$context) {
  if (empty($context['sandbox'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_entry'] = 0;
    // Assuming you implemented method for DatContainer class, that can
    // return number of records in the file and it's named rowCount.
    $context['sandbox']['max'] = $o_dat->rowCount();

    // Store the source file in the context.
    $context['sandbox']['source_file'] = $o_dat;
  }
  // Process 5 entries per iteration.
  $limit = 5;

  // Assuming you have method getRows to get N number of entries starting from X entry.
  foreach ($o_dat->getRows($context['sandbox']['current_entry'], $limit) as $po_entry) {
    $context['message'] = "Processing: {$po_entry->s_name}";
    $o_node = _get_entry_node_by_id($ps_platform, $po_entry->s_id);
    $_SESSION['http_request_count']++;

    $context['sandbox']['progress']++;
    // Assuming you have number for entry within file.
    $context['sandbox']['current_entry'] = $po_entry->entry_number;
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }

}

So now instead of N queue items you would have only one item, that would get called recursively as long as $context['finished'] < 1.

For more information about $context variable please visit callback_batch_operation() page. It's preferable to use this variable to store information used across batch operation callbacks, rather than $_SESSION.