Files migration issue

Question

I am following this tutorial on how to migrate files from D7 into D8.

I have the migration setup with the following yml configuration:

uuid: a685a4e0-1022-4bf2-a787-f6f1bef587ea
langcode: en
status: true
dependencies: {  }
id: d8_files
class: null
field_plugin_method: null
cck_plugin_method: null
migration_tags:
  - 'Drupal 7'
  - Content
migration_group: D7-TO-D8
label: 'D7 to 8 Public files'
source:
  plugin: d7_file
  scheme: public
  constants:
    # source_base_path: 'sites/default/files/'
    # source_base_path: '/var/www/d8/sites/default/files/'
    # source_base_path: '/var/www/d8/sites/default/files/d7_download/page/files/'
    # source_base_path: '/sites/default/files/d7_download/page/files/'
    # source_base_path: '/var/www/d7/sites/default/files/page/files/'
    source_base_path: '/var/www/d7/sites/default/files/'
process:
  fid: fid
  filename: filename
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/source_base_path
        - filepath
    -
      plugin: urlencode
  uri:
    plugin: file_copy
    source:
      - '@source_full_path'
      - uri
  filemime: filemime
  status: status
  created: timestamp
  changed: timestamp
  uid: uid
destination:
  plugin: 'entity:file'
migration_dependencies: null

Migrate-status indicates I have over 2000 files detected in the D7 database ready for migration but when I try migrating it just reports 0 migrated content and errors, shown below:

My guess is the migration does not know where to find the images but I have tried a variety of paths to those image by altering the source_base_path value however I still get no migrated files.

Can anyone help, please?

UPDATE:

I found out more on the issue preventing the migration.

When I checked the logs for the d8_files migration, I found messages such as the following:

File'/var/www/d8/sites/default/files/d7_download//sites/default/files/field/image/imagefield_OvRwVF.jpeg' does not exist

The issue is how the module is creating the file url.

In the source section of my d8_file.yml, I had set the following:

source_base_path: '/var/www/d8/sites/default/files/d7_download/'

... which is the absolute path to the directory containing the files I wish to migrate. The migrate module is concatenating /sites/default/files to the end of the path which makes it wrong. I have tried doing things like navigating up the directory path by using ../../ in the source_base_path settings but it doesn't work ... it's just seen as a string.

Can anyone tell me how to correct this?

In the process section, I saw the following:

process:
  fid: fid
  filename: filename
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/source_base_path
        - filepath

I suspect that concat value is the problem. BUT, I have no idea what other value to set it to.

PS: Can anyone advice where is the system to find such information on any plugin ???

Answer 1

If you need to do replacements in the files path like "public://" and "private://" to the full path to your public or private files, instead of just giving the filepath as the source, make the necessary replacements before:

  source_public_absolute_path:
    -
      plugin: str_replace
      source: uri
      search: 'public://'
      replace: 'sites/default/files/'
  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/source_base_path
        - '@source_public_absolute_path'

Answer 2

You can find the active issue here on Drupal.org: https://www.drupal.org/project/drupal/issues/3123350

As a workaround you could run this in the source database copy to remove the prefix:

DELETE FROM variable WHERE name="file_private_path";

See the issue for further discussion and background.