0

I'm working on the MongoDB source plugin and I'm able to do the migration, but I have to make strange compromises. When defining the source fields, I have to set name the same as selector, like this.

fields:
  -
    name: _id
    label: 'Unique ID'
    selector: _id

That works, but if I decide to change field name and keep selector, the migration fails.

fields:
  -
    name: node_id
    label: 'Unique ID'
    selector: _id

I get this error.

'node_id' is defined as a source ID but has no value.

Why is that? Why is the source array accessed by name instead of by selector?

1
  • 1
    This highly depends on the source plugin implementation. For instance, in migrate_plus's JSON source, the selector is xpath to the data and name is the shorthand. However, for migrage_source_csv, the name must be the heading value. It also has label but no selector. So you'll have to dig into the source's implementation to see what properties it needs and what it does.
    – Joseph
    Nov 3, 2021 at 13:51

2 Answers 2

1

You can change the name of source fields, but you must not change the name of source fields used as source IDs.

source:
  fields:
    -
      name: _id
      label: 'Unique ID'
      selector: _id
  ids:
    _id:   <-- this is the problem, you can not rename source fields used here
      type: integer

On the first migration the module creates a mapping table migration_map_MIGRATION_ID for each remote source row <=> local entity. After that, you no longer can change the name of source fields used as source IDs.

The only way around this is a complete rollback, then manually deleting those migration tables (including table migration_message_MIGRATION_ID ) and then you can make a fresh start.

1
  • @hundri thanks for explaining this. However source parameter is not working at all, not even for other fields. From what Joseph said and from other source I think that source plugin it self has to implement that functionality. But problem is I can find and instructions/tutorial on how to do that.
    – MilanG
    Nov 4, 2021 at 8:48
0

Selectors are handled by data processor plugins. Initially I tried to use iterator object provided by MongoDB collection object. That works, but selectors are not supported. Then I saw how custom data processor is made in Migrate plus module, for parsing json and made one based on that. Basically I'm using collection iterator to get full json and then my custom data processor works with that json the similar way as Migrate Plus json data processor does.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.