2

I have a very simple taxonomy term migration (migrate module, that lets in duplicate terms for no reason I can figure out. I have tried explicitly setting allow_duplicates to False, even though that should be the default behavior, but nothing changes. The base migration that this migration inherits from just sets up a db connection object, sets up team description variables, and sets some configuration variables. It also sets up a few helper methods that are not used in this migration.

Here is the migration class:

class EnviroscopeKeywordMigration extends BaseEnviroscopeMigration
{
    public function __construct($arguments)
    {
        parent::__construct($arguments);

        $this->description = t('import Keyword data');
        $query = $this->getConnection()
            ->select('doc_keyword', 'k')
            ->fields('k', array('ID', 'NAME'));

        $this->dependencies = array("GeoArea");

        $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE));

        $this->destination = new MigrateDestinationTerm('keywords');
        $this->map = new MigrateSQLMap($this->machineName,
            array(
                'ID' => array(
                    'type' => 'int',
                    'length' => 11,
                    'not null' => TRUE,
                    'description' => 'Keyword ID',
                ),
            ),
            MigrateDestinationTerm::getKeySchema()
        );

        $this->addFieldMapping('name', 'NAME');

        $this->addUnmigratedDestinations(array(
            'description',
            'parent',
            'parent_name',
            'format',
            'weight',
            'path',
            'pathauto',
        ));
    }

}

Can somebody, PLEASE help me debug why this might be bringing in duplicate terms like this:

enter image description here

  • The file that you're migrating has these duplicates? – No Sssweat Mar 3 '16 at 12:28
  • I'm migrating from another MySQL database, it has duplicates in it. They should all be combined into one term per phrase/word and the the migration map all the source Ids should map to that single destination. – UltraBob Mar 3 '16 at 12:31
  • Too many for manual deletion? Is this other db a Drupal db? – No Sssweat Mar 3 '16 at 12:44
  • The other db is not a Drupal db, too many for manual deletion, and other migrations depend on those source Ids being mapped properly so the other data gets associated with the right terms. – UltraBob Mar 3 '16 at 12:46
2

I ended up biting the bullet and writing it myself. I still don't know what was causing it to not work in the first place, but I replicated the desired behavior with this helper method in the migration class:

public function check_for_existing_term($taxonomy, $term) {
    $query = db_select('taxonomy_vocabulary', 'v')
        ->condition('machine_name', $taxonomy);
    $query->innerJoin('taxonomy_term_data', 'td', 'v.vid=td.vid');
    $new_tid = $query->fields('td', array('tid'))
        ->condition('td.name', $term)
        ->execute()
        ->fetchField();
    return ($new_tid) ? $new_tid : False;
}

In the constructor of the migration class I set $this->taxonomy = 'keywords'; and I used that in $this->destination = new MigrateDestinationTerm($this->taxonomy);, then I added the following prepareRow:

    public function prepareRow($row)
    {
        $new_tid = $this->check_for_existing_term($this->taxonomy, $row->NAME);
        if ($new_tid) {
            $this->map->saveIDMapping($row, array($new_tid),
                MigrateMap::STATUS_IGNORED, MigrateMap::ROLLBACK_DELETE);
            $this->rollbackAction = MigrateMap::ROLLBACK_DELETE;
            return FALSE;
        } else {
            return TRUE;
        }
    }

This tries to find a matching term in the database for each row. If it finds it, it adds an entry in the map table and then returns FALSE so the migration won't do anything further for that row. If it is a new term it returns True, and the migration continue happily, adding in the new term.

  • That's essentially what I was saying. – Kevin Mar 5 '16 at 17:27
  • Except that your solution would have returned without writing anything into the migration map for duplicate terms. Later when I ran the migration of nodes that need keywords associated with them The first instance of each would be the only one that got a term mapped. – UltraBob Mar 5 '16 at 23:34
0

There are two ways to eliminate duplicates. First the select-statement should have the "DISTINCT" qualification. Second there is a dedupe method for the proces. Online documentation will help you further, i.e. http://blog.studio.gd/node/5. But you must use the database-wrapper supplied by Drupal. Also adjust settings.php with a $databases 'migrate', 'default' declaration.

  • Neither of those do what I want. dedupe ensures that each term comes in as unique, which is the behavior I am trying to avoid. Select distinct will mean that there is only a migrate map mapping for one of the instances of the term. What needs to happen is for there to be a row in the migrate map for every source, but for all identical sources to map to the same destination. This enables future node migrations to properly map terms onto the nodes. – UltraBob Mar 8 '16 at 0:23
  • High UltraBob, seems you have to apply the first rule of migration. Ensure that your source data is correct. So first dedub your source and make sure there is only one occurrence per term is present. Then do the migration. Otherwise you have to do all sorts of acrobatic during your migration. – Jan Baelemans Mar 8 '16 at 9:02
  • My source data is correct for the system I'm migrating from, but is in a state that needs to be improved since it currently provides no utility. The answer I wrote here addresses that. – UltraBob Mar 8 '16 at 9:05
0

One way I used to do this was use the built in prepareRow method. I would query and check for term exists by name for the vocabulary I am importing to. If I got a result, I would return false, which prevents that record from being migrated. Taxonomy terms are tricky.

In this case it does not sound like you have unique terms in the source database, so I'm not sure how you can flatten multiples down to just one term while preserving relational data on content that used those terms (who's ID has changed in Drupal).

  • That is no good because all the source Ids need to be mapped to the right destination terms so that the migrations that depend on this one will be associated with the right terms. I may end up having to write a method to manually do the duplication check and mapping that migrate's documentation and behavior in other migrations suggest it should be doing by default. – UltraBob Mar 3 '16 at 12:50
  • The source terms have unique Ids, just not unique terms so what needs to happen is for the migration map to be written one row per source but many source Ids mapping to the same destination tid – UltraBob Mar 3 '16 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.