0

How can I create individual migration groups - one for each XLM file, without having to write a new migration class for each file?

I have dozens of XML files waiting to be imported into Drupal using the Migrate module and API.

Currently, all my migrations are registered under a single group called Questions as shown below: enter image description here That single group of 222 imports represents 5/6 of my XML files. If I import the pending dozens of XML files, I will have upwards of 10,000 imports.

What I would rather have is each XML file having it's own individual group. That way I can revert a single XML file where necessary instead of all them.

Can anyone help me with this scenario?

  • Commenting after I answer: why would you need each migration in its own group? If the Questions group has, say, 10 migrations in it, each representing a single XML file, you can migrate or roll back either the whole group or an individual migration within the group. I'm not sure what splitting your one group into a bunch of smaller groups does for you. – mounty Aug 24 '17 at 2:46
  • That questions group combines multiple xml files. My purpose is to have a distinct migration group for each xml file. That way I can roll back an individual xml file where necessary instead of rolling back all the files – sisko Aug 24 '17 at 19:45
  • So in your ideal scenario, how many migration configurations would there be per group? Just one for the one file? – mounty Aug 24 '17 at 20:02
  • Ah, gotcha. My migration knowledge is limited to D8. Sorry. – mounty Aug 24 '17 at 20:20
0

Update: this answer is probably fine for D8 but not D7.

Using the migrate_plus module, I was able to specify shared configuration for about a dozen different feeds that needed to be given separate taxonomies upon import based on which organization's media I was importing. I created a group with my source and process configuration, then created individual importers for each feed with just the configuration that was specific to that feed. If I wanted to import them all at once I could run drush mi --group=<group_name>, and if I wanted to roll back a single feed I could run drush migrate-rollback <migration_name>. The only tedious part for me was creating a dozen nearly-identical migrations.

So, for example, here's a group definition for an Twitter importer:

id: twitter_importer
source_type: twitter
shared_configuration:
  source:
    plugin: url
    data_fetcher_plugin: http
    data_parser_plugin: json
    authentication:
      plugin: oauth2
      base_uri: 'https://api.twitter.com'
      token_url: /oauth2/token
      grant_type: client_credentials
    fields:
      -
        name: id
        selector: id_str
      -
        name: text
        selector: text
      -
        name: timestamp
        selector: created_at
    ids:
      id:
        type: string
  process:
    title:
      plugin: substr
      source: caption
      start: 0
      length: 254
    type:
      plugin: default_value
      default_value: twitter_feed_item
    # etc. etc.

Then a few migrations like this:

id: tw_dept_a
dependencies:
  - migrate
  - migrate_plus
migration_group: twitter_importer
source:
  # this changes from migration to migration
  urls: 'https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=tmountjr&count=10'
process:
  field_source:
    # this also changes; works because the group's shared config specifies the plugin
    default_value: tmountjr
destination:
  plugin: 'entity:node'

It's annoying setting it all up on the frontend but after the third one you get pretty quick at copy/pasting the template and replacing one or two variables.

So I guess to answer your question:

How can I create individual migration groups - one for each XLM file, without having to write a new migration class for each file?

I'm not sure you can get around that...but you can abstract a lot of the repeated configuration out of each individual file using shared configurations. You can maybe further refine the process by having a template file with placeholder variables, duplicating the file, and using sed on the new file to replace filenames:

cp migration_template.yml migrate_plus.migration.migrate_xml1.yml
sed -i "s/filenameplaceholder/file1\.xml/g" migrate_plus.migration.migrate_xml1.yml
  • I'm very sorry. I should have indicated I am working in Drupal 7. Thanks for your answer and I apologise for not specifying the version. – sisko Aug 24 '17 at 20:17
0

The answer to your question is to use registerMigration as described here: https://www.drupal.org/node/1824884

Once upon a time the D7 migration classes supported a DynamicMigration class which is now obsolete: https://www.drupal.org/node/2471206

But the idea is that per XML file you would do something like:

foreach ($xml_files as $file) {
  MigrationBase::registerMigration(
  'ExampleUserMigration',
  'ExampleUser',
   array(
      'group_name' => 'example',
      'default_uid' => 1, 
      'my_source_file' => $file)
  );
}

And your migration would examine the passed in custom my_source_file migration argument and set the migration to use that specific file.

The above code could simply be a simple php script you call using Drush for example to "register" your migrations.

EDIT

Dynamic migrations are still supported somewhat in D8 via: https://www.drupal.org/docs/8/api/migrate-api/building-dynamic-migrations

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.