0

My Drupal migration yaml files are not being imported in the correct execution order.

I have 8 migration files, most that depend on another file being executed first. I'm using a Google Spreadsheet as the source, and targeting Drupal 8.9+. I have tagged all my dependencies, and all was going well with my test imports (using drush migrate:import --tag="Rooster Business") and the corresponding rollback commands. But at some point, after adding a few fields, the migrations began to fail because they were being executed out of order (particularly my business_locations and business_offers migrations).

Here's the exact error:

[error]  Migration rooster_node_business_offers did not meet the requirements. 
Missing migrations rooster_node_business_location. requirements: rooster_node_business_location.

The new fields themselves don't fail, so it's not that (I wouldn't think). I think I have correctly set up the dependencies in all the files, but they are NOT being respected (at least not completely).

I have tried the following:

  • Alphabetizing all the migration id's and filenames (to order them by desired import execution order).
  • Dropping all migration tables from the database and doing full cache refresh
  • Changing the last file I want executed (rooster_node_business_offers) to use "optional" dependencies instead of "required." (This causes the failure to go away, but then some of the desired field values are missing from the migration).
  • Uninstalling and reinstalling the entire custom migration module.
  • I've tried just migrating the "last" migration with drush migrate:import rooster_node_business_offers --execute-dependencies command which should in theory execute all the files in order as they all depend in a chain on each other. No dice. I get the same error: [error] Migration rooster_node_business_offers did not meet the requirements...

None of these attempts have solved the problem. I have a theory that it has something to do with the fields names of the sources (because I think I may have changed a few of them, and have definitely added quite a few). But that still doesn't account for why the migrations are not respecting the dependency order.

Bottom line: I have two questions:

  1. Why are the last two migration files (rooster_node_business_location & rooster_node_business_offers) not being executed in order by dependency (and how can I fix this)?
  2. If that is not an easy to answer question, what is the logic of the "default" ordering, and how can I game that logic so I can still get my migration files to execute in the desired order?

Here is the order I'd LIKE to have the migrations to execute in:

rooster_file_bg_photo
rooster_file_logo
rooster_users
rooster_file_logo_to_media
rooster_file_bg_photo_to_media
rooster_node_parent_business
rooster_node_business_location
rooster_node_business_offers

And here is the order that they are actually being migrated in (same as the order displayed after entering the drush ms command:

  Rooster (Rooster)   rooster_file_bg_photo            Idle
  Rooster (Rooster)   rooster_file_logo                Idle
  Rooster (Rooster)   rooster_users                    Idle
  Rooster (Rooster)   rooster_file_logo_to_media       Idle
  Rooster (Rooster)   rooster_file_bg_photo_to_media   Idle
  Rooster (Rooster)   rooster_node_parent_business     Idle
  Rooster (Rooster)   rooster_node_business_offers     Idle
  Rooster (Rooster)   rooster_node_business_location   Idle

I've made a repository of my exact migration files (and I've only redacted the Google Spreadsheet URLs). Here are all the migration files on Github.

Thanks in advance for your help!

  • A migration can't be optional if you use its ID in this migration with migration_lookup. And you have a circular dependency between your last two migrations. – Hudri Aug 14 at 7:36
  • It was my understanding that migrations that need data from a migration that come afterwards will create stubs and then insert that data once the migration becomes available. As a matter of fact, this has worked in my migrations when the migrations were executed in order. Do you know how I can at least get the migrations to execute in the order I want? – Doomd Aug 14 at 15:39
  • I just commented out the migration_lookup fields that required subsequent migrations. But now my nodes don't have references to each other. Is there a way to write a few additional YAML files that import/process these reference fields (this time, after the requisite migrations) without overwriting the values of all the other previously imported fields? For example, Can Migration #1A with Fields A-C be imported, then Migration #2 which depends on Migration #1A, THEN Migration #1B with field D (which depends on Migration #2)? Basically, I'd like some migration files to UPDATE, not overwrite. – Doomd Aug 14 at 16:22
  • 1
    Simply specify nid (or uid key in case of user entity) in the process section of migration #1B, e.g. through a migration_lookup or entity_lookup, then you can update an existing entity. – Hudri Aug 14 at 16:37
  • 1
    So, I did some more research. As I suspected, you ARE allowed to reference migration_lookups for migrations that are imported afterwards. BUT, there is a bug. The id of the migration file needed in subsequent references has to be in REVERSE alphabetical order: drupal.org/project/drupal/issues/3024634 – Doomd Aug 14 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.