Migration of Users very slow with d2d

Question

I'm wondering what could be causing my migration to run so slowly. I'm using dev versions of migrate and d2d as I always have without issue. There are over 120,000 users and the first 2000-9000 import within a few seconds.

After that it only processes 2 at a time (about every 30 seconds).

d2d code:

  $common_arguments = array(
    'source_connection' => 'legacy',
    'source_version' => 6,
  );

  $arguments = $common_arguments + array(
    'description' => t('Migration of users from Drupal 6'),
    'machine_name' => 'User',
  );

  Migration::registerMigration('DrupalUser6Migration', $arguments['machine_name'], $arguments);

include file:

class User extends DrupalUser6Migration {
  public function __construct(array $arguemnts) {
    parent::__construct($arguemnts);

    $this->addFieldMapping('roles', 'roles')
      ->sourceMigration('Roles');
  }
}

The legacy database the data is coming from is on my local machine on the same SSD the new site is on. Any ideas on why this is running so slow? At the rate it's going out it will take over a month to import all of the users...

Answer 1

In settings.php set:

$conf['password_count_log2'] = 1;

You will speed up how drupal handles user passwords: The problems reported here: https://www.drupal.org/node/2200181 ... seem to match yours, first 2000 in X seconds second 2000 in X.Y seconds etc...

Also, in php.ini set:

max_execution_time to 0 (for unlimited) or a high number like 3000 rather than the default 30.

NB. Be sure to return the execution time to its default setting after the migration.

Answer 2

CR47. I had same problem, I had to migrate 300 000 of users and batch migration became very slow. I used drush migrate commands and migration was rather fast (approximately 1000 of users per minute). I used

drush migrate-import User

where User is a machine name of my migration.

Also you can use

drush migrate-import User --feedback="1000 items"

In this case you will get feedback when 1000 of users will be migrated.