0

I have been trying to attach media entities to node entities in Drupal 8 with a custom migration but despite numerous attempts and countless hours reviewing forums i'm stumped - I'm desperately hoping someone can help explain where I might be going wrong.

Steps I have taken:

  1. Created a custom migration to import files (Using Migrate Plus & Migrate Source CSV) - successful

  2. Custom migration to import files into Drupal media entities (image bundle) - successful

Now that I have the media imported, I am trying to reference it so I can look to ditch the normal default field_image.

CSV format:

fid, uid, filename, filepath, uri, filemime, filesize, status, timestamp, uuid, alt 

As the files are now media elements I know I can't reference them by file ID (FID), presumably it would be the media ID (MID) but I have no idea how to get this.

I tried to use the migrate_lookup to reference the existing migrations but during import i'm not getting any error messages which makes it hard to tell whether i'm targeting the right thing or not.

I have also tried to add a default value to test a single value but this also didn't work.

id: news_nodes
label: News Articles
migration_group: custom

dependencies:
  enforced:
    module:
      - migrate_custom

source:
  plugin: news_nodes
  node_type: article
  target: db_migration

destination:
  plugin: entity:node
  default_bundle: article

process:
  nid: nid
  vid: vid
  uid:
    plugin: default_value
    default_value: 1
  langcode:
    plugin: default_value
    source: language
    default_value: en
  title: title
  uid: node_uid
  status: status
  created: created
  changed: changed
  promote: promote
  sticky: sticky
  revision_uid: revision_uid
  revision_log: log
  revision_timestamp: timestamp
  body/format:
    plugin: static_map
    bypass: true
    source: body_format
    map:
      1: plain_text
      2: restricted_html
      3: full_html
      4: full_html
  body/value: body_value
  body/summary: body_summary
  field_tags: tags

  field_image: image
  field_image: 
    -
      plugin: migration_lookup
      migration: import_files
      source: filename
    -
      plugin: skip_on_empty
      method: process

  field_media_image:
      plugin: sub_process
      source: field_media_image
      process: 
        target_id:
          - 
            plugin: migration_lookup
            migration: import_media_files
            source: fid
            no_stub: true
          -  
            plugin: skip_on_empty
            method: process

  # field_media_image/target_id:
  #   -
  #     plugin: migration_lookup
  #     migration: import_files
  #     source: image_fid
  #   -
  #     plugin: media_generate
  #     destinationField: field_media_image
  #     imageAltSource: 
  #       plugin: default_value
  #       default_value: 'Featured image'
  #     imageTitleSource:
  #       plugin: default_value
  #       default_value: 'Featured image'
  #   -  
  #     plugin: skip_on_empty
  #     method: process


  field_news_companies: company_tags
  field_news_companies:
    plugin: sub_process
    source: field_company_companies
    process:
      target_id: tid
migration_dependencies:
  required:
    - custom_user
    - import_files
    - import_media_files
    - upgrade_d7_taxonomy_term_company_companies
    - upgrade_d7_taxonomy_term_tags

The commented out section was trying this: https://drupal.stackexchange.com/a/249401/94547 however I get 'Cannot load a NULL ID. in assert()' error.

The node entity i'm trying to import to is the default article content type.

  • Default image field - field_image
  • Name of media field - field_media_image

I really would like to understand what i'm missing. I'm new to migrations so any advice on how to help would be appreciated!

EDIT

Having looked at a number of replies, I think we have narrowed down the issue to my source plugin plugin: news_nodes. which I am listing below.

The files/media have been imported using the CSV source but the nodes use custom PHP to query other custom fields which are required (taxonomy).

If I use the same source and plugin details as field_image in my field_media_image a media entity is referenced but not one that matches the file used in field_image. I'm guessing it's passing the file id and the media entity is handing back a media item with the same numeric id.

Below is the custom plugin. For all I know the way I have set this up entirely wrong and needs to be re-made - any advice would be great.

Full code available on Github: https://github.com/lmonk72/migrate_custom

<?php

namespace Drupal\migrate_custom\Plugin\migrate\source;

use Drupal\migrate\Row;
use Drupal\migrate\Plugin\migrate\source\SqlBase;

/**
 * @file
 * Contains \Drupal\migrate_custom\Plugin\migrate\source\Node.
 */

/**
 * Extract nodes from Drupal 7 database.
 *
 * @MigrateSource(
 *   id = "news_nodes"
 * )
 */
class NewsNodes extends SqlBase {

  /**
   * {@inheritdoc}
   */
  public function query() {
    // this queries the built-in metadata, but not the body, tags, or images.
    $query = $this->select('node', 'n')
      ->condition('n.type', 'article')
      ->fields('n', array(
        'nid',
        'vid',
        'type',
        'language',
        'title',
        'uid',
        'status',
        'created',
        'changed',
        'promote',
        'sticky',
      ));
    $query->orderBy('nid');
    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    $fields = $this->baseFields();
    $fields['body/format'] = $this->t('Format of body');
    $fields['body/value'] = $this->t('Full text of body');
    $fields['body/summary'] = $this->t('Summary of body');
    $fields['field_image/target_id'] = $this->t('Image');
    return $fields;
  }

  // /**
  //  * {@inheritdoc}
  //  */
  public function prepareRow(Row $row) {
    $nid = $row->getSourceProperty('nid');

    // body (compound field with value, summary, and format)
    $result = $this->getDatabase()->query('
      SELECT
        fld.body_value,
        fld.body_summary,
        fld.body_format
      FROM
        {field_data_body} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      $row->setSourceProperty('body_value', $record->body_value );
      $row->setSourceProperty('body_summary', $record->body_summary );
      $row->setSourceProperty('body_format', $record->body_format );
    }

    // taxonomy term IDs
    // (here we use MySQL's GROUP_CONCAT() function to merge all values into one row.)
    $result = $this->getDatabase()->query('
      SELECT
        GROUP_CONCAT(fld.field_tags_tid) as tids
      FROM
        {field_data_field_tags} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      if (!is_null($record->tids)) {
        $row->setSourceProperty('tags', explode(',', $record->tids) );
      }
    }

    // Company taxonomy term IDs
    // (here we use MySQL's GROUP_CONCAT() function to merge all values into one row.)
    $result = $this->getDatabase()->query('
      SELECT
        GROUP_CONCAT(fld.field_company_companies_tid) as tids
      FROM
        {field_data_field_company_companies} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    foreach ($result as $record) {
      if (!is_null($record->tids)) {
        $row->setSourceProperty('company_tags', explode(',', $record->tids) );
      }
    }

    $result = $this->getDatabase()->query('
      SELECT
        fld.field_image_fid,
        fld.field_image_alt,
        fld.field_image_title,
        fld.field_image_width,
        fld.field_image_height
      FROM
        {field_data_field_image} fld
      WHERE
        fld.entity_id = :nid
    ', array(':nid' => $nid));
    // Create an associative array for each row in the result. The keys
    // here match the last part of the column name in the field table. 
     $image = [];

    foreach ($result as $record) {
      $image[] = [
        'target_id' => $record->field_image_fid,
        'alt' => $record->field_image_alt,
        'title' => $record->field_image_title,
        'width' => $record->field_image_width,
        'height' => $record->field_image_height,
      ];
    }
    $row->setSourceProperty('image', $image);


    return parent::prepareRow($row);
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['nid']['type'] = 'integer';
    $ids['nid']['alias'] = 'n';
    return $ids;
  }

  /**
   * {@inheritdoc}
   */
  public function bundleMigrationRequired() {
    return FALSE;
  }

  /**
   * {@inheritdoc}
   */
  public function entityTypeId() {
    return 'node';
  }

  /**
   * Returns the user base fields to be migrated.
   *
   * @return array
   *   Associative array having field name as key and description as value.
   */
  protected function baseFields() {
    $fields = array(
      'nid' => $this->t('Node ID'),
      'vid' => $this->t('Version ID'),
      'type' => $this->t('Type'),
      'title' => $this->t('Title'),
      'format' => $this->t('Format'),
      'teaser' => $this->t('Teaser'),
      'uid' => $this->t('Authored by (uid)'),
      'created' => $this->t('Created timestamp'),
      'changed' => $this->t('Modified timestamp'),
      'status' => $this->t('Published'),
      'promote' => $this->t('Promoted to front page'),
      'sticky' => $this->t('Sticky at top of lists'),
      'language' => $this->t('Language (fr, en, ...)'),
    );
    return $fields;
  }
}
?>
  • ids: [fid] this is PHP array syntax, not YAML syntax. Try with the same syntax as used in the keys: section directly above (with dash and indent) – Hudri Feb 18 at 16:39
  • Thanks for the quick reply! I'll look at trying that as soon as possible. Just to clarify, are you saying that when I have updated the syntax of [fid], that when I run the migration of the final file (news_nodes) the source will find it? On it's own, the import_media_files (which runs first) works successfully in adding the media to the media content only, not the node entity. I'll try screenshot a few bits to explain the current logic and to provide more insight into the issue. – lawrence_monk Feb 18 at 21:26
  • Where is fid coming from in the news_nodes migration? Unless I am getting yaml-blindness, it is not declared as a source column in your csv, so you will be feeding the migration_lookup plugin a load of nothings? – stevekeiretsu Feb 19 at 0:20
  • Does the migration_lookup exclusively look back at the previous import and use those values as keys? I've made some changes but it still isn't picking it up. I tried Hudri's suggestion to change the keys and the files imported correctly but I still can't seem to get the node entity to find them - issue must be somewhere in the news_node plugin - repo here: github.com/lmonk72/migrate_custom any suggestions? – lawrence_monk Feb 20 at 13:27
0

I have migrated images a few times, at first I used the same approach as you:

  1. Create an image migration.
  2. Use migration_lookup on the node migration.

This works fine, my best guess is your node migration fails to identify the image id because it references the file id. Taking a look at your migrate_map_import_media_files table should help you clarify things a bit.

That said, if you just need to get the job done, I would suggest you give migrate_file a try.

You can use it to migrate everything at once, so you wouldn't need the extra migrations.

Example usage:

source:
  constants:
    images_destination: 'public://images/'
  ...

process:
  field_media_image:
    -
      plugin: explode
      delimiter: ,
      source: images
    -
      plugin: image_import
      destination: 'constants/images_destination'
      title: !file
      alt: title
      skip_on_error: true

You only need the explode if you have a coma separated list of image paths, like I did. If you have only one file you can remove it.

|improve this answer|||||
  • Yes, you need to use migration_lookup and lookup the SOURCE ID of what they were, not their Drupal entity media IDs that were created. – Kevin Feb 20 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.