0

I'm creating a module to batch import/update nodes in Drupal 7 based on input files that can be XML or other formats. Worst case scenario is they are like ~10-20MB and can contain 10-20k entries.

Currently, the module works fine for small files (~2k entries), but I get timeouts during initialization (before the actual batch job) for bigger files. The input file parsing is not the problem, it works fine and this is an example table of parsing times:

ENTRIES  PARSING (s)  FORMAT
     63        0.056  xml
    544        0.144  xml
   1706        0.202  xml
   2741        0.940  xml  <- batch job doesn't start
   3398        0.419  txt  <- batch job doesn't start
   7916        8.726  xml  <- batch job doesn't start
   8607       29.734  xml  <- batch job doesn't start

This is the relevant part of my module's code:

function batch_import($ps_platform, $ps_input_file) {
    # I've tried to change timeout limit with no luck        
    set_time_limit(500);

    # Parsing the input file to my own class. The time spent for
    # different size files can be in the table above.
    $o_dat = new DatContainer($ps_input_file);

    # [1/2] Preparing the batch job array we will output
    #---------------------------------------------------
    $a_batch_job = array(
        'title' => t('Importing dat file'),
        'init_message' => t('Beginning dat import process'),
        'operations' => array(),
        'finished' => 'batch_finished',
        );

    # [2/2] Populate the operations sub-array with all the jobs we want to do
    # -----------------------------------------------------------------------
    foreach ($o_dat as $o_entry) {
        $a_batch_job['operations'][] = array('process_entry', array($ps_platform, $o_entry));
    }

    batch_set($a_batch_job);
}

The actual function to process each individual task doesn't do anything by now, it simply prints the progress.

function process_entry($ps_platform, $po_entry, &$context) {
    $context['message'] = "Processing: {$po_entry->s_name}";
    $o_node = _get_entry_node_by_id($ps_platform, $po_entry->s_id);
    $_SESSION['http_request_count']++;
}

Does anybody see where is the problem? After checking many examples for batch api, I'm quite frustrated because my code seems to be correct.

Regards and thanks for your time.

2

The timeout happens because of large amount of operations within $a_batch_job['operations'] array. For every operation defined queue item is created and stored in the queue table of the database.

Check the _batch_populate_queue() method.

Workaround

Instead of defining Pre-set list of calls, you can take a route of Alterable list of calls. So in your case it could be like this:

function batch_import($ps_platform, $ps_input_file) {
  # Parsing the input file to my own class. The time spent for
  # different size files can be in the table above.
  $o_dat = new DatContainer($ps_input_file);

  # [1/2] Preparing the batch job array we will output
  #---------------------------------------------------
  $a_batch_job = array(
    'title' => t('Importing dat file'),
    'init_message' => t('Beginning dat import process'),
    // Pass our parsed file as a parameter to the batch operation callback.
    'operations' => array('process_entry', array($o_dat, $ps_platform)),
    'finished' => 'batch_finished',
  );

  batch_set($a_batch_job);
}

And your process_entry() would look like this:

function process_entry($o_dat, $ps_platform, &$context) {
  if (empty($context['sandbox'])) {
    $context['sandbox']['progress'] = 0;
    $context['sandbox']['current_entry'] = 0;
    // Assuming you implemented method for DatContainer class, that can
    // return number of records in the file and it's named rowCount.
    $context['sandbox']['max'] = $o_dat->rowCount();

    // Store the source file in the context.
    $context['sandbox']['source_file'] = $o_dat;
  }
  // Process 5 entries per iteration.
  $limit = 5;

  // Assuming you have method getRows to get N number of entries starting from X entry.
  foreach ($o_dat->getRows($context['sandbox']['current_entry'], $limit) as $po_entry) {
    $context['message'] = "Processing: {$po_entry->s_name}";
    $o_node = _get_entry_node_by_id($ps_platform, $po_entry->s_id);
    $_SESSION['http_request_count']++;

    $context['sandbox']['progress']++;
    // Assuming you have number for entry within file.
    $context['sandbox']['current_entry'] = $po_entry->entry_number;
  }

  // Inform the batch engine that we are not finished,
  // and provide an estimation of the completion level we reached.
  if ($context['sandbox']['progress'] != $context['sandbox']['max']) {
    $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max'];
  }

}

So now instead of N queue items you would have only one item, that would get called recursively as long as $context['finished'] < 1.

For more information about $context variable please visit callback_batch_operation() page. It's preferable to use this variable to store information used across batch operation callbacks, rather than $_SESSION.

  • First of all, thanks for your very detailed answer. I wasn't aware that the batch tasks were stored in the DB and that adds delay to the process. After posting the question I took a workaround, dividing the long list of tasks into smaller chunks of ~20-40 elements. It's working fine but of course the progress bar just shows each of those chunks of tasks. I imagine that the situation with your approach would be similar and having just one big action would make the progress bar totally meaningless. Am I right? Regards and thanks again. – NoRandom Apr 6 '18 at 10:26
  • Remember this line of code? $context['finished'] = $context['sandbox']['progress'] / $context['sandbox']['max']; The bar will be shown based on it. E.g. the bigger the value, the closer progress bar to the end. Yet you are right,the message below progress bar "Performing 1 out of 1" will make no sense. However, $context['message'] = "Processing: {$po_entry->s_name}"; will appear above the progress bar. That's how you will be able to determine the current progress, along with percentages. – Sardis Apr 6 '18 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.