Overriding migration source URL with hook_migration_plugins_alter

Question

In D8, using Migrate Plus (8.x-4.0-beta3), i am pulling in xml records from an external, non-Drupal API. I have an unwieldy source URL that needs to be passed dynamically to the migrate_plus migration YML file. That doesn't seem possible, but I have read in comments that you can alter the source URL of a migration with hook_migrations_plugins_alter. However when i do something like below, the URL does not change.

function my_module_migration_plugins_alter(array &$migrations){

  foreach ($migrations as $id => $migration) {
    if ($migration['id'] == 'my_migrate_id'){
      unset($migrations['source']['urls']);
      $migration['source']['urls'] = 'http://example_api';
    }
  }
}

I realize I may need to override the source plugin; if this is true what method do i use to alter the source url?

Answer 1

If you don't want the overridden config to be saved to the database you can override it on the fly with ConfigFactoryOverrideInterface. I'm storing the file path in State API storage via a custom file upload form.

my_module.services.yml

services:
  my_module.overrider:
    class: \Drupal\my_module\MyModuleOverrides
    tags:
      - {name: config.factory.override, priority: 5}

src/MyModuleOverrides.php

<?php

namespace Drupal\my_module;

use Drupal\Core\Cache\CacheableMetadata;
use Drupal\Core\Config\ConfigFactoryOverrideInterface;
use Drupal\Core\Config\StorageInterface;

/**
 * Overrides migration configuration.
 */
class MyModuleOverrides implements ConfigFactoryOverrideInterface {

  /**
   * {@inheritdoc}
   */
  public function loadOverrides($names) {
    $overrides = array();
    $state = \Drupal::state();
    $overrides['migrate_plus.migration.my_migration']['source']['path'] = $state->get('my_module.my_migration_source_uri');
    return $overrides;
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheSuffix() {
    return 'MyModuleOverrider';
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheableMetadata($name) {
    return new CacheableMetadata();
  }

  /**
   * {@inheritdoc}
   */
  public function createConfigObject($name, $collection = StorageInterface::DEFAULT_COLLECTION) {
    return NULL;
  }

}

Answer 2

Your approach should work, as others have said you will need to clear cache. However there is a problem with your implementation. You're overriding the url on the $migration variable but never setting that back to the $migrations array.

Try this:

function my_module_migration_plugins_alter(array &$migrations){
  $migrations['my_migrate_id']['source']['urls'] = 'http://example_api';
}

Answer 3

There is an issue/patch for the Migrate Plus module, which solves this common problem on a generic YAML basis. It allows to use a PHP callback function as source for the URLs.

In your migrate YAML:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls:
    callback: my_module_migrate_urls

In your custom module file:
(The current migration is passed as parameter to the callback)

use Drupal\migrate\Plugin\MigrateIdMapInterface
 
function my_module_migrate_urls(MigrationInterface $migration): array {
      $ids = my_module_get_all_ids();
      $urls = [];
      foreach ($ids as $id) {
        $urls[] = "https://example.com/jsonapi/node/article/{$id}";
      }
      return $urls;
}

For more information see the linked issue, it is documented quite well.

Answer 4

unset($migrations['source']['urls']);

You've got a typo here - it should be $migration, not $migrations.

Also note that plugin definitions are cached - you may need to do a cache rebuild after making changes to your alter hook for them to take effect.

Answer 5

I was actually running into the same issue until this morning. The source URLs are stored within the Config state (which is in the database).

I successfully altered the URL by doing the following:

function my_module_migration_plugins_alter(&$migrations) {
    foreach ($migrations as $id => $migration) {
        if ($migration['id'] == 'my_migrate_id'){
            $config = \Drupal::service('config.factory')->getEditable('migrate_plus.migration.'.$migration['id']);

            $config->set('source.urls', 'http://example_api');
            $config->save();
        }
    }
}

Answer 6

In order to have dynamic URLs but keep all configuration in the YML file rather that scattered in alter hooks, we can use a custom source plugin to replace placeholders in the URL, and use standard Migrate processors to provide the value for the URL.

For example to import a list of persons that have their birthday today:

source:
  plugin: processable_url
    data_fetcher_plugin: http
    data_parser_plugin: json
    urls:
      - 'https://mysource.com/api/persons?birthdate={{ today }}'
    url_processors:
      - key: today
        process:
          plugin: default_value
          default_value: 'today'
      - key: today
        process:
          plugin: strtotime
      - key: today
        process:
          plugin: format_date
          from_format: 'U'
          to_format: 'Y-m-d'

ProcessableUrl migrate source plugin:

<?php

declare(strict_types = 1);

namespace Drupal\mymodule\Plugin\migrate\source;

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\migrate\MigrateExecutable;
use Drupal\migrate\Plugin\MigratePluginManagerInterface;
use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate\Row;
use Drupal\migrate_plus\DataParserPluginInterface;
use Drupal\migrate_plus\Plugin\migrate\source\Url;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Provides a migration source for a URL with processable placeholders.
 *
 * Replaces Jinja style {{ placeholders }}. Useful for REST APIs that require
 * dynamic values in the URL.
 *
 * Make sure to start with a `default_value` processor, or create your own
 * processor plugin that provides the required value.
 *
 * @MigrateSource(
 *   id = "processable_url"
 * )
 */
class ProcessableUrl extends Url implements ContainerFactoryPluginInterface {

  /**
   * The URL processors.
   *
   * @var array
   */
  protected array $urlProcessors = [];

  /**
   * Constructs a new ProcessableUrl migrate source plugin.
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, protected MigratePluginManagerInterface $migratePluginManager) {
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);

    $this->urlProcessors = $configuration['url_processors'] ?? [];
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL) {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $migration,
      $container->get('plugin.manager.migrate.process'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function getDataParserPlugin(): DataParserPluginInterface {
    if (!isset($this->dataParserPlugin)) {
      $configuration = ['urls' => $this->processUrls()] + $this->configuration;
      $this->dataParserPlugin = \Drupal::service('plugin.manager.migrate_plus.data_parser')->createInstance($this->configuration['data_parser_plugin'], $configuration);
    }
    return $this->dataParserPlugin;
  }

  /**
   * Processes the source URLs.
   *
   * @return array
   *   The processed URLs.
   */
  protected function processUrls(): array {
    $placeholders = $this->processPlaceholders();
    return array_map(function (string $url) use ($placeholders): string {
      foreach ($placeholders as $placeholder => $value) {
        $url = preg_replace('/\{\{\s*' . $placeholder . '\s*\}\}/', $value, $url);
      }
      return $url;
    }, $this->sourceUrls);
  }

  /**
   * Processes the placeholders.
   *
   * @return array
   *   The processed placeholders.
   */
  protected function processPlaceholders(): array {
    // @todo Doesn't support row based processors.
    $executable = new MigrateExecutable($this->migration);
    $row = new Row();

    $placeholders = [];
    foreach ($this->urlProcessors as $definition) {
      $key = $definition['key'];
      $plugin_definition = $definition['process'];
      $plugin = $this->migratePluginManager->createInstance($plugin_definition['plugin'], $plugin_definition);
      $placeholders[$key] = $plugin->transform($placeholders[$key] ?? NULL, $executable, $row, $key);
    }

    return $placeholders;
  }

}

And a simple StringToTime processor:

<?php

declare(strict_types = 1);

namespace Drupal\mymodule\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Converts a string representation of a date to a Unix timestamp.
 *
 * @MigrateProcessPlugin(
 *   id = "strtotime"
 * )
 */
class StringToTime extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    return strtotime($value);
  }

}