Error importing csv with batch API

Question

I'm using this code in my batch job:

<?php

function my_module_csv_import() {

  $folder = "csv";
  $files = scandir($folder);

  // define batch array structure
  // NOTE: minimal parameters defined to simplify code
  $batch = array(
    'title' => t('Reading File'),
    'operations' => array(
      array(
        '_my_module_batch_read', array($files_array),
        ),
        ),
        'finished' => '_my_module_batch_finished',
        );

        // set batch
        batch_set($batch);

        // process batch
        batch_process('');

        return '';

}


function _my_module_batch_read($files, &$context) {

  variable_set('site_offline', '1');

  // define batch limit
  $batch_limit = 500000;

  // assume the batch process has not completed
  $context['finished'] = 0;

  foreach($files as $csv_file_path) {

    // open the file for reading
    $file_handle = fopen($csv_file_path, 'r');

    if ($file_handle){
      // check if file pointer position exists in the sandbox, and jump to location in file
      if ($context['sandbox']['file_pointer_position']) {
        fseek($file_handle, $context['sandbox']['file_pointer_position']);
      }

      // loop through the file and stop at batch limit
      for ($i = 0; $i < $batch_limit; $i++) {

        // get file line as csv
        $csv_line = fgetcsv($file_handle);

        // NOTE: at this point, do what ever you'd like with the CSV array data!
        if (is_array($csv_line) && isset($csv_line[0]) && isset($csv_line[1])) {
          // DO STUFF
        } else {
          dpm("PROBLEM READING LINES");
        }

        // retain current file pointer position
        $context['sandbox']['file_pointer_position'] = ftell($file_handle);

        // check for EOF
        if (feof($file_handle)) {
          // complete the batch process
          $context['finished'] = 1;

          // end loop
          break;
        }
      }
    }
  }

  $context['finished'] = 1;

}
?>

it works fine, but sometimes it breaks.
The $csv_line results to have PROBLEM READING LINES and I get
warning: htmlspecialchars() [function.htmlspecialchars]: Invalid multibyte sequence in argument.
I can't figure out how this could be possible, since the csv is correct.
Can anyone point me in some solution?

Answer 1

Found the error: the problem is that I'm not resetting $context['sandbox']['file_pointer_position'].
This causes batch pointer start reading the CSV file in the wrong position.
To correct this, it's enough to add

$context['sandbox']['file_pointer_position'] = 0;

to the check for EOF

if (feof($file_handle)) {
    // complete the batch process
    $context['sandbox']['file_pointer_position'] = 0;
    $context['finished'] = 1;
    // end loop
    break;
}

Answer 2

The Drupal 6 Feeds module may be better suited for this task that a custom module. I've only used the D7 version myself, but I believe it supports Batch processing out of the box. It seems to have a decent API as well that you may be able to integrate with your module.

You may want to verify your CSV with a tool like Flat File Checker to ensure the issue isn't with the format of the document.