Migrate multiple migrations into same node

Question

I need to do a migration from multiple endpoints into the same node. All endpoints provide the full list of rows/nodes, but their data need to be mapped into different fields of the same node and into different translations (so not similar to this question, where one endpoint is the complete list and the second is the single row data).

Here is what I've got so far:

First migration, create the nodes in default language

id: kognitiv_rooms_base
label: Rooms Step 1 (room data in german and available rates)
migration_group: kognitiv
migration_tags:
  - rooms
  - masterdata
status: true

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: https://www.example.com/rooms_endpoint_one.json?lang=de
  item_selector: /result/rooms
  track_changes: true
  fields:
    -
      name: code
      label: 'Kognitiv room code'
      selector: code
    -
      name: title
      label: 'Room name'
      selector: title
    -
      name: description
      label: 'Room description'
      selector: description
    -
      name: min_occupancy
      label: 'Minimum room occupancy'
      selector: min_occupancy
    -
      name: std_occupancy
      label: 'Standard room occupancy'
      selector: std_occupancy
    -
      name: max_occupancy
      label: 'Maximum room occupancy'
      selector: max_occupancy
  ids:
    code:
      type: string

destination:
  plugin: entity:node

process:
  type:
    plugin: default_value
    default_value: room
  title: title
  langcode:
     plugin: default_value
     default_value: de
  uid:
    plugin: default_value
    default_value: 1
  sticky:
    plugin: default_value
    default_value: 0
  promote:
    plugin: default_value
    default_value: 0
  short_description: description
  long_description: description
  room_min_occupancy: min_occupancy
  room_std_occupancy: std_occupancy
  room_max_occupancy: max_occupancy
  remote_datasource:
    plugin: default_value
    default_value: kognitiv
  remote_id: code

dependencies:
  enforced:
    module:
      - kognitiv_importer

Second migration, fill additional fields in existing nodes

id: kognitiv_rooms_ammenities
label: Rooms Step 2 (ammenities and room size)
migration_group: kognitiv
migration_tags:
  - rooms
  - masterdata
status: true

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: https://www.example.com/rooms_endpoint_two.json
  item_selector: /facility/rooms
  track_changes: true
  fields:
    -
      name: code
      label: 'Kognitiv room code'
      selector: code
    -
      name: size
      label: 'Room size'
      selector: /types/0/size
  ids:
    code:
      type: string

destination:
  plugin: entity:node

process:
  type:
    plugin: default_value
    default_value: room
  nid:
    plugin: migration_lookup
    migration: kognitiv_rooms_base
    source: code
  uid:
    plugin: default_value
    default_value: 1
  roomsize: size

migration_dependencies:
  required:
    - kognitiv_rooms_base

dependencies:
  enforced:
    module:
      - kognitiv_importer

Third migration, translate the nodes

id: kognitiv_rooms_en
label: Rooms Step 3 (english translation)
migration_group: kognitiv
migration_tags:
  - rooms
  - masterdata
  - english
status: true

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: https://www.example.com/rooms_endpoint_one.json?lang=en
  item_selector: /result/rooms
  track_changes: true
  fields:
    -
      name: code
      label: 'Kognitiv room code'
      selector: code
    -
      name: title
      label: 'Room name'
      selector: title
    -
      name: description
      label: 'Room description'
      selector: description
  ids:
    code:
      type: string

destination:
  plugin: entity:node
  translations: true

process:
  type:
    plugin: default_value
    default_value: room
  langcode:
     plugin: default_value
     default_value: en
  content_translation_source:
    plugin: default_value
    default_value: de
  nid:
    plugin: migration_lookup
    source: code
    migration: kognitiv_rooms_base
  title: title
  uid:
    plugin: default_value
    default_value: 1
  short_description: description
  long_description: description

migration_dependencies:
  required:
    - kognitiv_rooms_base
    - kognitiv_rooms_ammenities

dependencies:
  enforced:
    module:
      - kognitiv_importer

I've got the idea with setting nid with migrate_lookup from those examples, using it for translations.

My problem is that only the first migration works. Both the 2nd and 3rd migration fail because of a "Integrity constraint violation: 1062 Duplicate entry '100000130' for key 'PRIMARY': INSERT INTO {node} (nid, vid, type, uuid, langcode)...". The nid value is correct, but I guess it should be an update instead of an insert.

How can I force Migrate to update an existing node? The 2nd query should update the node in the same language, the 3rd query should add/update the english translation of the node.

I'm on Drupal 8.4, using Migrate Plus 4.0

Answer 1

The code above works, including updating existing nodes in the same language and adding/updating translations. After installing the module on a fresh site, everything worked. Obviously during the numerous iterations of developing, uninstalling and reinstalling, some invisible old data/cache/whatever was left in a rollback run and caused the error.