D2D file migration cannot processes a number of rows?

Question

In my Drupal 6 site I have 147490 files in the database. On my Drupal 7 site I am using D2D migrate module to import them. The number of Items it sees is exactly the same, 147490. However, it will not processes 5683 of them. 112229 import successfully and 29578 fail (because the file no longer exists on the server, that's fine). When re-running the migration, it will not processes the remaining 5683.

Here is my migration class:

  $file_arguments = $common_arguments + array(
    'machine_name' => 'Files',
    'description' => t('Migration of files from Drupal 6'),
    'user_migration' => 'User',
    'default_uid' => 1,
    'source_dir' => 'http://lcl.legacy-d6.com/sites/default/files',
    'destination_dir' => 'public://',
  );

  Migration::registerMigration('Files', $file_arguments['machine_name'], $file_arguments);

I am then extending the class with an include: file_migration.inc

class Files extends DrupalFile6Migration {
  public function __construct(array $arguemnts) {
    parent::__construct($arguemnts);

  }

  public function prepareRow($row) {
    if (parent::prepareRow($row) === FALSE) {
      return FALSE;
    }

    $filepath  = $row->filepath;
    $new_filepath = str_replace('/var/www/html/liv-dir/production/', '', $filepath);
    $new_filepath = str_replace('/var/www/html/lcl.legacy-d6.com/htdocs/', '', $new_filepath);
    $row->filepath = $new_filepath;

    return TRUE;
  }

}

EDIT: I was able to create a snippet that finds all of the missing Ids however I was not able to figure out why they are not being processed.

    set_time_limit(0);
    $unprocessed_ids = array();
    db_set_active('legacy');
    $get_fids_query = Database::getConnection('legacy')->select('files', 'fi');
    $get_fids_query->fields('fi', array('fid'));
    $result = $get_fids_query->execute();
    while ($fid = $result->fetchField()) {
      db_set_active();
      $check_migration_query = Database::getConnection('default')->select('migrate_map_files', 'mfi');
      $check_migration_query->fields('mfi', array('sourceid1'));
      $check_migration_query->condition('sourceid1', $fid, '=');
      $check_migration_result = $check_migration_query->execute();
      $check_migration = $check_migration_result->fetchField();
      if ($check_migration === FALSE || is_null($check_migration) || !isset($check_migration)) {
        $unprocessed_ids[] = $fid;
      }
      db_set_active('legacy');
    }
  }

UPDATE: The only fix I could find that actually worked was waiting for the migration to run its course and then rollback a small number of rows, and then run through the migration again with --update flag set. Depending on the envioronment I was running on (local vs pantheon) I would have to do this between 5-20 times.

I still want to know why this was happening.

Answer 1

I just faced the same issue, and have a suspicion what the cause may be, though have not tested it.

I ran a migration of files in manual batches, e.g.

drush migrate-import MyMigration --limit="100 items"

several times. After the last batch (for which I did not specify an item limit, i.e. it should have imported all remaining unprocessed items), migrate-status still showed almost 900 unprocessed items. All calls thereafter to migrate-import would process 0 items, even with the --update flag -- which I thought should have either imported or updated (accordingly) ALL items...

Just as you did, I then queried for the source IDs it would not process. Upon inspection I noticed that large sets of them had the same exact timestamp (specifically, file_managed.timestamp column in the database). I recalled that Migrate can use a specified field to store a highwater mark for each migration, such that future import runs will only process items whose value of that field is greater than the stored highwater mark from previous runs. So for files, the timestamp value is used by default (per Migrate D2D implementation -- see DrupalFile7Migration::__construct()), and the query is ordered by timestamp ASC to cooperate with this.

I realized that the items it would not process were definitely older than the stored highwater timestamp (which is that of the newest item imported). Why then did Migrate not process those in an earlier batch, based on the query's order-by? Well... if many of my files had the same exact timestamp, the results of the query were not guaranteed to be in the same order across different runs. So if I was working with a limit of 100 items per run, but the results were not consistently ordered, it's possible some items never ended up falling within the batch (range of results) that was actually being imported at the time.

I'm not entirely sure if this could account for nearly 900 unprocessed images, but it's a thought.

Workaround

You can import items regardless of their status vs. the highwater mark by using the --idlist argument. So since you've queried to find the source IDs that won't process, just pass them as a comma-delimited string and I think it should work no matter what. Example:

drush migrate-import MyMigration --idlist="538,2891,485"

Actual fix? (untested)

If my suspicion is correct, then probably the solution is to order by not only timestamp, but additionally by fid, which is guaranteed unique, and thus the results order will be exactly the same every time. In your migration subclass, override query() and after calling the parent implementation, add $query->orderBy('fid', 'ASC'). This sets secondary ordering.