Migrating revisions from external data source

Question

We are trying to migrate content from an external data source into Drupal 8 using the core migrate modules. This is a stripped down example of the data being migrated:

id     resourceId     articleId     version     title          body
---------------------------------------------------------------------------
1      10             10            1           Page title     Page content
2      10             10            1.1         Page title     Page content
3      10             10            1.2         Page title     Page content
4      11             11            1           Page title     Page content
5      11             11            1.1         Page title     Page content

The id field is the unique identifier, and is the field set in the getIds() method, when as you can see the resourceId and articleId are more a preferred representation for a node id to be migrated into Drupal.

From my understanding, after reading through the migrate_example modules, the query() method must only return a single row; so when our migration runs, all these records are migrated as new nodes when in the previous database they were just revisions of one piece of content.

If we are intended to only return a single row in the query() method, what is the best way to only migrate the most recent version of each piece of content?

My first attempt was to try and achieve this in the processRow() method, but then if the migration is going through the table row by row, then how can you remove/update a previous row if the next row is a more recent version?

We have an option to just run our own script on the table first, to filter out all most recent versions and then run the migrate script on that, but I would like to know if there is a way to achieve this using the core modules in Drupal.

Thanks in advance.

Answer 1

Yeah, this part is not documented at all, I had now also problems with it, so here is how I solved it:
When you want to migrate a content that has revisions, you need to migrate the first version with the entity:node plugin, then you migrate the next versions with the entity_revision:node. You don't do anything else, just change the destination plugin and when you migrate the revisions, you set the nid value to the first version of the node. So you keep the body and the file and other fields, you just add the nid field, so the revision plugin will know where to put the content. That's it at least from the yml part.
From the source, you have two one for the node plugin where you list only the first version of your content that you want to migrate and one for the revisions where you list everything EXCEPT the first version also, you have a new column which contains the first version's ID for every revision's it's own.

If you migrate the current publish one into the node table, and you migrate the rest into the revision, the IDs, the order of the revision and some other things will be messed up. At least my case was that.

By code (not 100% complette, just main parts):

migrate_plus.migration.node.yml:

id: my_node
source:
  plugin: my_node
process:
  title: title
  body: body
  langcode:
    plugin: default_value
    default_value: "und"
  status:
    plugin: default_value
    default_value: 1
  #other fields except nid.
destination:
  plugin: entity:node

Query for the source plugin (id = my_node):
SELECT id, title, body FROM content WHERE version = 1
in getIds() the id is the only id.

migrate_plus.migration.node_revision.yml:

id: my_node_revision
source:
  plugin: my_node_revision
process:
  nid:
    plugin: migration_lookup
    migration:
     - my_node
     - my_node_revision
    source: first_version
  title: title
  body: body
  langcode:
    plugin: default_value
    default_value: "und"
  status:
    plugin: default_value
    default_value: 1
  #other fields.
destination:
  plugin: entity_revision:node

Query for the source plugin (id = my_node_revision):

SELECT first_ver.id AS first_version, c.id, c.title, c.body FROM content c
  INNER JOIN (
    SELECT * FROM content c
    WHERE version = 1
  ) first_ver ON c.resourceId = first_ver.resourceId AND c.articleId = first_ver.articleId
WHERE c.version <> 1

in getIds() the id is the only id.

The above example will give new id's for the content.

EDIT: note that the example in this answer is a migration from confluence database into drupal, so therefore slightly different than a normal drupal-drupal migration. Also, if you look in the entity_revision:node plugin, which is located here: core/modules/migrate/src/Plugin/migrate/destination/EntityRevision.php you will notice that you have two ways of doing this.
A) Providing revision id.
B) Not providing revision id.

When you don't provide the revision ID, then as you add new revisions, it will always make the latest added revision the latest content version. (the example yml and source are like this above)

When you provide a revision ID, then you first need to add the current version into the node table, then add all the revisions and provide the revision ID, in that case, it won't make the last added revision as the latest content version.