Migrate as separate files

Question

How can I create individual migration groups - one for each XLM file, without having to write a new migration class for each file?

I have dozens of XML files waiting to be imported into Drupal using the Migrate module and API.

Currently, all my migrations are registered under a single group called Questions as shown below: That single group of 222 imports represents 5/6 of my XML files. If I import the pending dozens of XML files, I will have upwards of 10,000 imports.

What I would rather have is each XML file having it's own individual group. That way I can revert a single XML file where necessary instead of all them.

Can anyone help me with this scenario?

Answer 1

Update: this answer is probably fine for D8 but not D7.

Using the migrate_plus module, I was able to specify shared configuration for about a dozen different feeds that needed to be given separate taxonomies upon import based on which organization's media I was importing. I created a group with my source and process configuration, then created individual importers for each feed with just the configuration that was specific to that feed. If I wanted to import them all at once I could run drush mi --group=<group_name>, and if I wanted to roll back a single feed I could run drush migrate-rollback <migration_name>. The only tedious part for me was creating a dozen nearly-identical migrations.

So, for example, here's a group definition for an Twitter importer:

id: twitter_importer
source_type: twitter
shared_configuration:
  source:
    plugin: url
    data_fetcher_plugin: http
    data_parser_plugin: json
    authentication:
      plugin: oauth2
      base_uri: 'https://api.twitter.com'
      token_url: /oauth2/token
      grant_type: client_credentials
    fields:
      -
        name: id
        selector: id_str
      -
        name: text
        selector: text
      -
        name: timestamp
        selector: created_at
    ids:
      id:
        type: string
  process:
    title:
      plugin: substr
      source: caption
      start: 0
      length: 254
    type:
      plugin: default_value
      default_value: twitter_feed_item
    # etc. etc.

Then a few migrations like this:

id: tw_dept_a
dependencies:
  - migrate
  - migrate_plus
migration_group: twitter_importer
source:
  # this changes from migration to migration
  urls: 'https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=tmountjr&count=10'
process:
  field_source:
    # this also changes; works because the group's shared config specifies the plugin
    default_value: tmountjr
destination:
  plugin: 'entity:node'

It's annoying setting it all up on the frontend but after the third one you get pretty quick at copy/pasting the template and replacing one or two variables.

So I guess to answer your question:

How can I create individual migration groups - one for each XLM file, without having to write a new migration class for each file?

I'm not sure you can get around that...but you can abstract a lot of the repeated configuration out of each individual file using shared configurations. You can maybe further refine the process by having a template file with placeholder variables, duplicating the file, and using sed on the new file to replace filenames:

cp migration_template.yml migrate_plus.migration.migrate_xml1.yml
sed -i "s/filenameplaceholder/file1\.xml/g" migrate_plus.migration.migrate_xml1.yml

Answer 2

The answer to your question is to use registerMigration as described here: https://www.drupal.org/node/1824884

Once upon a time the D7 migration classes supported a DynamicMigration class which is now obsolete: https://www.drupal.org/node/2471206

But the idea is that per XML file you would do something like:

foreach ($xml_files as $file) {
  MigrationBase::registerMigration(
  'ExampleUserMigration',
  'ExampleUser',
   array(
      'group_name' => 'example',
      'default_uid' => 1, 
      'my_source_file' => $file)
  );
}

And your migration would examine the passed in custom my_source_file migration argument and set the migration to use that specific file.

The above code could simply be a simple php script you call using Drush for example to "register" your migrations.

EDIT

Dynamic migrations are still supported somewhat in D8 via: https://www.drupal.org/docs/8/api/migrate-api/building-dynamic-migrations