2

In D8, using Migrate Plus (8.x-4.0-beta3), i am pulling in xml records from an external, non-Drupal API. I have an unwieldy source URL that needs to be passed dynamically to the migrate_plus migration YML file. That doesn't seem possible, but I have read in comments that you can alter the source URL of a migration with hook_migrations_plugins_alter. However when i do something like below, the URL does not change.

function my_module_migration_plugins_alter(array &$migrations){

  foreach ($migrations as $id => $migration) {
    if ($migration['id'] == 'my_migrate_id'){
      unset($migrations['source']['urls']);
      $migration['source']['urls'] = 'http://example_api';
    }
  }
}

I realize I may need to override the source plugin; if this is true what method do i use to alter the source url?

6 Answers 6

4

If you don't want the overridden config to be saved to the database you can override it on the fly with ConfigFactoryOverrideInterface. I'm storing the file path in State API storage via a custom file upload form.

my_module.services.yml

services:
  my_module.overrider:
    class: \Drupal\my_module\MyModuleOverrides
    tags:
      - {name: config.factory.override, priority: 5}

src/MyModuleOverrides.php

<?php

namespace Drupal\my_module;

use Drupal\Core\Cache\CacheableMetadata;
use Drupal\Core\Config\ConfigFactoryOverrideInterface;
use Drupal\Core\Config\StorageInterface;

/**
 * Overrides migration configuration.
 */
class MyModuleOverrides implements ConfigFactoryOverrideInterface {

  /**
   * {@inheritdoc}
   */
  public function loadOverrides($names) {
    $overrides = array();
    $state = \Drupal::state();
    $overrides['migrate_plus.migration.my_migration']['source']['path'] = $state->get('my_module.my_migration_source_uri');
    return $overrides;
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheSuffix() {
    return 'MyModuleOverrider';
  }

  /**
   * {@inheritdoc}
   */
  public function getCacheableMetadata($name) {
    return new CacheableMetadata();
  }

  /**
   * {@inheritdoc}
   */
  public function createConfigObject($name, $collection = StorageInterface::DEFAULT_COLLECTION) {
    return NULL;
  }

}
3

Your approach should work, as others have said you will need to clear cache. However there is a problem with your implementation. You're overriding the url on the $migration variable but never setting that back to the $migrations array.

Try this:

function my_module_migration_plugins_alter(array &$migrations){
  $migrations['my_migrate_id']['source']['urls'] = 'http://example_api';
}
1

There is an issue/patch for the Migrate Plus module, which solves this common problem on a generic YAML basis. It allows to use a PHP callback function as source for the URLs.

In your migrate YAML:

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls:
    callback: my_module_migrate_urls

In your custom module file:
(The current migration is passed as parameter to the callback)

use Drupal\migrate\Plugin\MigrateIdMapInterface
 
function my_module_migrate_urls(MigrationInterface $migration): array {
      $ids = my_module_get_all_ids();
      $urls = [];
      foreach ($ids as $id) {
        $urls[] = "https://example.com/jsonapi/node/article/{$id}";
      }
      return $urls;
}

For more information see the linked issue, it is documented quite well.

0
unset($migrations['source']['urls']);

You've got a typo here - it should be $migration, not $migrations.

Also note that plugin definitions are cached - you may need to do a cache rebuild after making changes to your alter hook for them to take effect.

3
  • Oh, but actually the unset line doesn't really matter, you're setting the right thing below. The cache rebuild comment stands.
    – Mike Ryan
    Jun 8, 2018 at 15:50
  • Thank you for answering this. Good to know i am in the right direction. After running drush cr multiple times, however, the migration still pulls in the URL that is in the migration YML file... Any other suggestions?
    – kb_lib
    Jun 15, 2018 at 20:24
  • I don't have any other ideas as to why it's not working for you - my next step would be, well, stepping through in a debugger.
    – Mike Ryan
    Jun 17, 2018 at 15:11
0

I was actually running into the same issue until this morning. The source URLs are stored within the Config state (which is in the database).

I successfully altered the URL by doing the following:

function my_module_migration_plugins_alter(&$migrations) {
    foreach ($migrations as $id => $migration) {
        if ($migration['id'] == 'my_migrate_id'){
            $config = \Drupal::service('config.factory')->getEditable('migrate_plus.migration.'.$migration['id']);

            $config->set('source.urls', 'http://example_api');
            $config->save();
        }
    }
}
0

In order to have dynamic URLs but keep all configuration in the YML file rather that scattered in alter hooks, we can use a custom source plugin to replace placeholders in the URL, and use standard Migrate processors to provide the value for the URL.

For example to import a list of persons that have their birthday today:

source:
  plugin: processable_url
    data_fetcher_plugin: http
    data_parser_plugin: json
    urls:
      - 'https://mysource.com/api/persons?birthdate={{ today }}'
    url_processors:
      - key: today
        process:
          plugin: default_value
          default_value: 'today'
      - key: today
        process:
          plugin: strtotime
      - key: today
        process:
          plugin: format_date
          from_format: 'U'
          to_format: 'Y-m-d'

ProcessableUrl migrate source plugin:

<?php

declare(strict_types = 1);

namespace Drupal\mymodule\Plugin\migrate\source;

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\migrate\MigrateExecutable;
use Drupal\migrate\Plugin\MigratePluginManagerInterface;
use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate\Row;
use Drupal\migrate_plus\DataParserPluginInterface;
use Drupal\migrate_plus\Plugin\migrate\source\Url;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**
 * Provides a migration source for a URL with processable placeholders.
 *
 * Replaces Jinja style {{ placeholders }}. Useful for REST APIs that require
 * dynamic values in the URL.
 *
 * Make sure to start with a `default_value` processor, or create your own
 * processor plugin that provides the required value.
 *
 * @MigrateSource(
 *   id = "processable_url"
 * )
 */
class ProcessableUrl extends Url implements ContainerFactoryPluginInterface {

  /**
   * The URL processors.
   *
   * @var array
   */
  protected array $urlProcessors = [];

  /**
   * Constructs a new ProcessableUrl migrate source plugin.
   */
  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, protected MigratePluginManagerInterface $migratePluginManager) {
    parent::__construct($configuration, $plugin_id, $plugin_definition, $migration);

    $this->urlProcessors = $configuration['url_processors'] ?? [];
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL) {
    return new static(
      $configuration,
      $plugin_id,
      $plugin_definition,
      $migration,
      $container->get('plugin.manager.migrate.process'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function getDataParserPlugin(): DataParserPluginInterface {
    if (!isset($this->dataParserPlugin)) {
      $configuration = ['urls' => $this->processUrls()] + $this->configuration;
      $this->dataParserPlugin = \Drupal::service('plugin.manager.migrate_plus.data_parser')->createInstance($this->configuration['data_parser_plugin'], $configuration);
    }
    return $this->dataParserPlugin;
  }

  /**
   * Processes the source URLs.
   *
   * @return array
   *   The processed URLs.
   */
  protected function processUrls(): array {
    $placeholders = $this->processPlaceholders();
    return array_map(function (string $url) use ($placeholders): string {
      foreach ($placeholders as $placeholder => $value) {
        $url = preg_replace('/\{\{\s*' . $placeholder . '\s*\}\}/', $value, $url);
      }
      return $url;
    }, $this->sourceUrls);
  }

  /**
   * Processes the placeholders.
   *
   * @return array
   *   The processed placeholders.
   */
  protected function processPlaceholders(): array {
    // @todo Doesn't support row based processors.
    $executable = new MigrateExecutable($this->migration);
    $row = new Row();

    $placeholders = [];
    foreach ($this->urlProcessors as $definition) {
      $key = $definition['key'];
      $plugin_definition = $definition['process'];
      $plugin = $this->migratePluginManager->createInstance($plugin_definition['plugin'], $plugin_definition);
      $placeholders[$key] = $plugin->transform($placeholders[$key] ?? NULL, $executable, $row, $key);
    }

    return $placeholders;
  }

}

And a simple StringToTime processor:

<?php

declare(strict_types = 1);

namespace Drupal\mymodule\Plugin\migrate\process;

use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;

/**
 * Converts a string representation of a date to a Unix timestamp.
 *
 * @MigrateProcessPlugin(
 *   id = "strtotime"
 * )
 */
class StringToTime extends ProcessPluginBase {

  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    return strtotime($value);
  }

}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.