How can I reference previously imported media entities?

Question

I have been trying to attach media entities to node entities in Drupal 8 with a custom migration but despite numerous attempts and countless hours reviewing forums i'm stumped - I'm desperately hoping someone can help explain where I might be going wrong.

Steps I have taken:

1. Created a custom migration to import files (Using Migrate Plus & Migrate Source CSV) - successful

2. Custom migration to import files into Drupal media entities (image bundle) - successful

Now that I have the media imported, I am trying to reference it so I can look to ditch the normal default field_image.

CSV format:

fid, uid, filename, filepath, uri, filemime, filesize, status, timestamp, uuid, alt 

As the files are now media elements I know I can't reference them by file ID (FID), presumably it would be the media ID (MID) but I have no idea how to get this.

I tried to use the migrate_lookup to reference the existing migrations but during import i'm not getting any error messages which makes it hard to tell whether i'm targeting the right thing or not.

I have also tried to add a default value to test a single value but this also didn't work.

id: news_nodes
label: News Articles
migration_group: custom

dependencies:
  enforced:
    module:
      - migrate_custom

source:
  plugin: news_nodes
  node_type: article
  target: db_migration

destination:
  plugin: entity:node
  default_bundle: article

process:
  nid: nid
  vid: vid
  uid:
    plugin: default_value
    default_value: 1
  langcode:
    plugin: default_value
    source: language
    default_value: en
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  body/format:
    plugin: static_map
    bypass: true
    source: body_format
    map:
      1: plain_text
      2: restricted_html
      3: full_html
      4: full_html
  body/value: body_value
  body/summary: body_summary
  field_tags: tags

  field_image: image
  field_image: 
    -
      plugin: migration_lookup
      migration: import_files
      source: filename
    -
      plugin: skip_on_empty
      method: process

  field_media_image:
      plugin: sub_process
      source: field_media_image
      process: 
        target_id:
          - 
            plugin: migration_lookup
            migration: import_media_files
            source: fid
            no_stub: true
          -  
            plugin: skip_on_empty
            method: process

  # field_media_image/target_id:
  #   -
  #     plugin: migration_lookup
  #     migration: import_files
  #     source: image_fid
  #   -
  #     plugin: media_generate
  #     destinationField: field_media_image
  #     imageAltSource: 
  #       plugin: default_value
  #       default_value: 'Featured image'
  #     imageTitleSource:
  #       plugin: default_value
  #       default_value: 'Featured image'
  #   -  
  #     plugin: skip_on_empty
  #     method: process


  field_news_companies: company_tags
  field_news_companies:
    plugin: sub_process
    source: field_company_companies
    process:
      target_id: tid
migration_dependencies:
  required:
    - custom_user
    - import_files
    - import_media_files
    - upgrade_d7_taxonomy_term_company_companies
    - upgrade_d7_taxonomy_term_tags

The commented out section was trying this: https://drupal.stackexchange.com/a/249401/94547 however I get 'Cannot load a NULL ID. in assert()' error.

The node entity i'm trying to import to is the default article content type.

• Default image field - field_image
• Name of media field - field_media_image

I really would like to understand what i'm missing. I'm new to migrations so any advice on how to help would be appreciated!

EDIT

Having looked at a number of replies, I think we have narrowed down the issue to my source plugin plugin: news_nodes. which I am listing below.

The files/media have been imported using the CSV source but the nodes use custom PHP to query other custom fields which are required (taxonomy).

If I use the same source and plugin details as field_image in my field_media_image a media entity is referenced but not one that matches the file used in field_image. I'm guessing it's passing the file id and the media entity is handing back a media item with the same numeric id.

Below is the custom plugin. For all I know the way I have set this up entirely wrong and needs to be re-made - any advice would be great.

Full code available on Github: https://github.com/lmonk72/migrate_custom

<?php

namespace Drupal\migrate_custom\Plugin\migrate\source;

use Drupal\migrate\Row;
use Drupal\migrate\Plugin\migrate\source\SqlBase;

/**
 * @file
 * Contains \Drupal\migrate_custom\Plugin\migrate\source\Node.
 */

/**
 * Extract nodes from Drupal 7 database.
 *
 * @MigrateSource(
 *   id = "news_nodes"
 * )
 */
class NewsNodes extends SqlBase {

  /**
   * {@inheritdoc}
   */
  public function query() {
    // this queries the built-in metadata, but not the body, tags, or images.
    $query = $this->select('node', 'n')
      ->condition('n.type', 'article')
      ->fields('n', array(
        'nid',
        'vid',
        'type',
        'language',
        'title',
        'uid',
        'status',
        'created',
        'changed',
        'promote',
        'sticky',
      ));
    $query->orderBy('nid');
    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    $fields = $this->baseFields();
    $fields['body/format'] = $this->t('Format of body');
    $fields['body/value'] = $this->t('Full text of body');
    $fields['body/summary'] = $this->t('Summary of body');
    $fields['field_image/target_id'] = $this->t('Image');
    return $fields;
  }

  // /**
  //  * {@inheritdoc}
  //  */
  public function prepareRow(Row $row) {
    $nid = $row->getSourceProperty('nid');

    // body (compound field with value, summary, and format)
    $result = $this->getDatabase()->query('
      SELECT
        fld.body_value,
        fld.body_summary,
        fld.body_format
      FROM
        {field_data_body} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      $row->setSourceProperty('body_value', $record->body_value );
      $row->setSourceProperty('body_summary', $record->body_summary );
      $row->setSourceProperty('body_format', $record->body_format );
    }

    // taxonomy term IDs
    // (here we use MySQL's GROUP_CONCAT() function to merge all values into one row.)
    $result = $this->getDatabase()->query('
      SELECT
        GROUP_CONCAT(fld.field_tags_tid) as tids
      FROM
        {field_data_field_tags} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      if (!is_null($record->tids)) {
        $row->setSourceProperty('tags', explode(',', $record->tids) );
      }
    }

    // Company taxonomy term IDs
    // (here we use MySQL's GROUP_CONCAT() function to merge all values into one row.)
    $result = $this->getDatabase()->query('
      SELECT
        GROUP_CONCAT(fld.field_company_companies_tid) as tids
      FROM
        {field_data_field_company_companies} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      if (!is_null($record->tids)) {
        $row->setSourceProperty('company_tags', explode(',', $record->tids) );
      }
    }

    $result = $this->getDatabase()->query('
      SELECT
        fld.field_image_fid,
        fld.field_image_alt,
        fld.field_image_title,
        fld.field_image_width,
        fld.field_image_height
      FROM
        {field_data_field_image} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    // Create an associative array for each row in the result. The keys
    // here match the last part of the column name in the field table. 
     $image = [];

    foreach ($result as $record) {
      $image[] = [
        'target_id' => $record->field_image_fid,
        'alt' => $record->field_image_alt,
        'title' => $record->field_image_title,
        'width' => $record->field_image_width,
        'height' => $record->field_image_height,
      ];
    }
    $row->setSourceProperty('image', $image);


    return parent::prepareRow($row);
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['nid']['type'] = 'integer';
    $ids['nid']['alias'] = 'n';
    return $ids;
  }

  /**
   * {@inheritdoc}
   */
  public function bundleMigrationRequired() {
    return FALSE;
  }

  /**
   * {@inheritdoc}
   */
  public function entityTypeId() {
    return 'node';
  }

  /**
   * Returns the user base fields to be migrated.
   *
   * @return array
   *   Associative array having field name as key and description as value.
   */
  protected function baseFields() {
    $fields = array(
      'nid' => $this->t('Node ID'),
      'vid' => $this->t('Version ID'),
      'type' => $this->t('Type'),
      'title' => $this->t('Title'),
      'format' => $this->t('Format'),
      'teaser' => $this->t('Teaser'),
      'uid' => $this->t('Authored by (uid)'),
      'created' => $this->t('Created timestamp'),
      'changed' => $this->t('Modified timestamp'),
      'status' => $this->t('Published'),
      'promote' => $this->t('Promoted to front page'),
      'sticky' => $this->t('Sticky at top of lists'),
      'language' => $this->t('Language (fr, en, ...)'),
    );
    return $fields;
  }
}
?>

Answer 1

I have migrated images a few times, at first I used the same approach as you:

1. Create an image migration.
2. Use migration_lookup on the node migration.

This works fine, my best guess is your node migration fails to identify the image id because it references the file id. Taking a look at your migrate_map_import_media_files table should help you clarify things a bit.

That said, if you just need to get the job done, I would suggest you give migrate_file a try.

You can use it to migrate everything at once, so you wouldn't need the extra migrations.

Example usage:

source:
  constants:
    images_destination: 'public://images/'
  ...

process:
  field_media_image:
    -
      plugin: explode
      delimiter: ,
      source: images
    -
      plugin: image_import
      destination: 'constants/images_destination'
      title: !file
      alt: title
      skip_on_error: true

You only need the explode if you have a coma separated list of image paths, like I did. If you have only one file you can remove it.

Answer 2

What I did that work:

  field_hero_image/target_id:
-
  plugin: migration_lookup
  migration: previous_migrate_media_image
  source: node_uid
-
  plugin: skip_on_empty
  method: row
  message: 'nid is missing'


migration_dependencies:
  required:
    - previous_migrate_user
    - previous_migrate_media_image
  optional: { }