Migration bringing in duplicate taxonomy terms

Question

I have a very simple taxonomy term migration (migrate module, that lets in duplicate terms for no reason I can figure out. I have tried explicitly setting allow_duplicates to False, even though that should be the default behavior, but nothing changes. The base migration that this migration inherits from just sets up a db connection object, sets up team description variables, and sets some configuration variables. It also sets up a few helper methods that are not used in this migration.

Here is the migration class:

class EnviroscopeKeywordMigration extends BaseEnviroscopeMigration
{
    public function __construct($arguments)
    {
        parent::__construct($arguments);

        $this->description = t('import Keyword data');
        $query = $this->getConnection()
            ->select('doc_keyword', 'k')
            ->fields('k', array('ID', 'NAME'));

        $this->dependencies = array("GeoArea");

        $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE));

        $this->destination = new MigrateDestinationTerm('keywords');
        $this->map = new MigrateSQLMap($this->machineName,
            array(
                'ID' => array(
                    'type' => 'int',
                    'length' => 11,
                    'not null' => TRUE,
                    'description' => 'Keyword ID',
                ),
            ),
            MigrateDestinationTerm::getKeySchema()
        );

        $this->addFieldMapping('name', 'NAME');

        $this->addUnmigratedDestinations(array(
            'description',
            'parent',
            'parent_name',
            'format',
            'weight',
            'path',
            'pathauto',
        ));
    }

}

Can somebody, PLEASE help me debug why this might be bringing in duplicate terms like this:

Answer 1

I ended up biting the bullet and writing it myself. I still don't know what was causing it to not work in the first place, but I replicated the desired behavior with this helper method in the migration class:

public function check_for_existing_term($taxonomy, $term) {
    $query = db_select('taxonomy_vocabulary', 'v')
        ->condition('machine_name', $taxonomy);
    $query->innerJoin('taxonomy_term_data', 'td', 'v.vid=td.vid');
    $new_tid = $query->fields('td', array('tid'))
        ->condition('td.name', $term)
        ->execute()
        ->fetchField();
    return ($new_tid) ? $new_tid : False;
}

In the constructor of the migration class I set $this->taxonomy = 'keywords'; and I used that in $this->destination = new MigrateDestinationTerm($this->taxonomy);, then I added the following prepareRow:

    public function prepareRow($row)
    {
        $new_tid = $this->check_for_existing_term($this->taxonomy, $row->NAME);
        if ($new_tid) {
            $this->map->saveIDMapping($row, array($new_tid),
                MigrateMap::STATUS_IGNORED, MigrateMap::ROLLBACK_DELETE);
            $this->rollbackAction = MigrateMap::ROLLBACK_DELETE;
            return FALSE;
        } else {
            return TRUE;
        }
    }

This tries to find a matching term in the database for each row. If it finds it, it adds an entry in the map table and then returns FALSE so the migration won't do anything further for that row. If it is a new term it returns True, and the migration continue happily, adding in the new term.

Answer 2

There are two ways to eliminate duplicates. First the select-statement should have the "DISTINCT" qualification. Second there is a dedupe method for the proces. Online documentation will help you further, i.e. http://blog.studio.gd/node/5. But you must use the database-wrapper supplied by Drupal. Also adjust settings.php with a $databases 'migrate', 'default' declaration.

Answer 3

One way I used to do this was use the built in prepareRow method. I would query and check for term exists by name for the vocabulary I am importing to. If I got a result, I would return false, which prevents that record from being migrated. Taxonomy terms are tricky.

In this case it does not sound like you have unique terms in the source database, so I'm not sure how you can flatten multiples down to just one term while preserving relational data on content that used those terms (who's ID has changed in Drupal).

Answer 4

Just in case someone else would have the same problem on a drupal 9, I developed a small module containing a migrate process for that: https://www.drupal.org/project/migrate_merge_duplicated_terms