4

I'm using this code in my batch job:

<?php

function my_module_csv_import() {

  $folder = "csv";
  $files = scandir($folder);

  // define batch array structure
  // NOTE: minimal parameters defined to simplify code
  $batch = array(
    'title' => t('Reading File'),
    'operations' => array(
      array(
        '_my_module_batch_read', array($files_array),
        ),
        ),
        'finished' => '_my_module_batch_finished',
        );

        // set batch
        batch_set($batch);

        // process batch
        batch_process('');

        return '';

}


function _my_module_batch_read($files, &$context) {

  variable_set('site_offline', '1');

  // define batch limit
  $batch_limit = 500000;

  // assume the batch process has not completed
  $context['finished'] = 0;

  foreach($files as $csv_file_path) {

    // open the file for reading
    $file_handle = fopen($csv_file_path, 'r');

    if ($file_handle){
      // check if file pointer position exists in the sandbox, and jump to location in file
      if ($context['sandbox']['file_pointer_position']) {
        fseek($file_handle, $context['sandbox']['file_pointer_position']);
      }

      // loop through the file and stop at batch limit
      for ($i = 0; $i < $batch_limit; $i++) {

        // get file line as csv
        $csv_line = fgetcsv($file_handle);

        // NOTE: at this point, do what ever you'd like with the CSV array data!
        if (is_array($csv_line) && isset($csv_line[0]) && isset($csv_line[1])) {
          // DO STUFF
        } else {
          dpm("PROBLEM READING LINES");
        }

        // retain current file pointer position
        $context['sandbox']['file_pointer_position'] = ftell($file_handle);

        // check for EOF
        if (feof($file_handle)) {
          // complete the batch process
          $context['finished'] = 1;

          // end loop
          break;
        }
      }
    }
  }

  $context['finished'] = 1;

}
?>

it works fine, but sometimes it breaks.
The $csv_line results to have PROBLEM READING LINES and I get
warning: htmlspecialchars() [function.htmlspecialchars]: Invalid multibyte sequence in argument.
I can't figure out how this could be possible, since the csv is correct.
Can anyone point me in some solution?

2

Found the error: the problem is that I'm not resetting $context['sandbox']['file_pointer_position'].
This causes batch pointer start reading the CSV file in the wrong position.
To correct this, it's enough to add

$context['sandbox']['file_pointer_position'] = 0;

to the check for EOF

if (feof($file_handle)) {
    // complete the batch process
    $context['sandbox']['file_pointer_position'] = 0;
    $context['finished'] = 1;
    // end loop
    break;
}
1

The Drupal 6 Feeds module may be better suited for this task that a custom module. I've only used the D7 version myself, but I believe it supports Batch processing out of the box. It seems to have a decent API as well that you may be able to integrate with your module.

You may want to verify your CSV with a tool like Flat File Checker to ensure the issue isn't with the format of the document.

  • what should I pass as argument? $file_handle? – pasine Feb 1 '12 at 9:26
  • Where is htmlspecialchars() being called from? I do not see it in your code example. Does your CSV contain non UTF8 characters or do your CSV filenames contain non UTF8 characters by chance? – Citricguy Feb 1 '12 at 10:56
  • not at all. The htmlspecialchars error is raised by Drupal in the bootstrap.inc. I'm trying to understand what's the problem. I've just discovered that the file is not correctly parsed. I've added an extra control on the array and I've realized that even the array sometimes is empty (so the lines are not properly read). – pasine Feb 1 '12 at 11:06
  • Given that the error is out of bootstrap.inc I'm going to re-build my answer. I don't think this is an issue with d6 core. Using a contributed, better supported module may be the easiest route to take. Is your CSV more than 64,000 lines? – Citricguy Feb 1 '12 at 11:41
  • Updated again. :) – Citricguy Feb 1 '12 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.