Migration taking increasingly long periods of time

Question

I have a migration and I've noticed that the code, through each batch operation, seems to process less and less items.

For example, the first few cycles, it'll say, "processed 600", and 600 or around there will be processed. However about 5-10 cycles in, and it will still say "processed 600", and it actually processes perhaps 100 or 200. This eventually stops the migration from happening, as it essentially grinds to a halt.

It was completing on my local machine and not the stage environment, which I initially thought to be due to some sort of environmental condition. Now I think that the cycles on my local were just large enough to allow everything to process before it grinded to a halt.

I'm not sure what is causing this, and if it's in this migration, or the migration that this migration is extending. I'll include the former, and if anyone needs the latter, let me know.

Note, there may be occasional instances where the words Company or Type are capitalized when they should not be - I just did a quick find/replace for sensitive data.

class CompanyTypeUserDataMigration extends CompanyCertBaseMigration {

public function __construct($arguments) {
    parent::__construct($arguments);

    $this->machineName = "CompanyTypeUserData";
    $this->systemOfRecord = Migration::DESTINATION;
    $this->group = MigrateGroup::getInstance("Company");
    $query = Database::getConnection('default', 'Type')
        ->select('Type')
        ->fields('Type', array('Name', 'Username', 'Address', 'City', 'State', 'Zipcode', 'Country', 'Phone', 'Notes', 'Email'));
    $query->addExpression('`Company Name`', 'Company');
    $query->addExpression('`Supervisor Email`', "supervisor_email");
    $query->addExpression('`Declined Recert Renewal`', "declined_recert");
    $query->isNotNull('Username');
    $this->source = new MigrateSourceSQL($query, array('status' => 'Status calculated based on various fields'));
    $this->destination = new MigrateDestinationUser();
    $this->map = new MigrateSQLMap($this->machineName, array(
            'Username' => array('type' => 'varchar',
                'length' => 50,
                'unsigned' => FALSE,
                'not null' => FALSE,
            ),
            'Email' => array('type' => 'varchar',
                'length' => 50,
                'not null' => FALSE,
            )
        ), MigrateDestinationUser::getKeySchema()
    );

    $this->addFieldMapping('name', 'Username');
    $this->addFieldMapping('field_company', 'Company');
    $this->addFieldMapping('field_supervisor_email', 'supervisor_email');
    $this->addFieldMapping('field_notes', 'Notes');
    $this->addFieldMapping('field_status', 'status');
    $this->addFieldMapping('field_full_name', 'Name');
    $this->addFieldMapping('uid', 'user_id');
}

public function prepareRow($row) {
    // Always include this fragment at the beginning of every prepareRow()
    // implementation, so parent classes can ignore rows.
    if (parent::prepareRow($row) === FALSE) {
        return FALSE;
    }
    if (empty($row->Username)) {
        return FALSE;
    }

    $uid = $this->find_user_id($row);
    if($uid === FALSE){
        return false;
    } else {
        $row->user_id = $uid;
        $user = user_load($uid);
        if(isset($user->name)){
            $row->Username = $user->name; //so we don't try to overwrite username permutations
        }
    }
}

}

Answer 1

What's the total number of nodes you're importing? I've hit this issue before using the D2D Migration module. I've found that D2D starts to slow down on consecutive batches when I'm importing in more than ~5K nodes. In scenarios where I've needed to import 10K-100K nodes, batches eventually slow down to where nothing is ever imported.

Using XHProf & XDebug, I traced down the issue to database objects consuming tons of memory & CPU to the point of stopping PHP execution. I've never been able solve the issue for importing over 10K nodes using D2D. Searches all over the internet have produced nothing. In many instances I've had to rely on custom PHP import code as since I could not use D2D.