I'm migrating lots of records from a non-Drupal site into Drupal 9. The records have file 'attachments,' and I want these to be media in the Drupal site.
I have all the files already on the server where they need to permanently live and have all the paths/URI's stored with all the other site data in CSV files.
Successful so far: I can migrate the URIs into file entities without errors, using the migrate_source_csv module (along with migrate_plus and migration_tools)
The media items have some meta data, so I need to migrate them in separately. This is where results have been interesting.
I can migrate the media in from the same source CSV file and use the migration_lookup plugin to create the media entities and relate them to their files.
I can also migrate the media in using entity_lookup plugin with very similar results.
In both cases, everything seems normal except that many thousands of file uses of generic.png are generated. Generic.png is not in my source data. I think it comes from the media module.
I can remove these uses from the file_usage table without any apparent negative consequences, but I'm uneasy doing that. This is a partial migration for test purposes, so... 12,552 items created. I hesitate to do the whole thing and move on to migrating in the 'nodes' until I understand what's going on here and potential consequences.
At this stage, I'm using a custom media type called 'document'
The file+media with dependency and migrate_lookup configuration files look like this:
The files file...
uuid: 1bcec3e7-0a49-4473-87a2-6dca09b91abjan-docf id: docfiles_import label: "Import files for doc media type" migration_group: docfilesmedia source: plugin: 'csv' path: '/srv/imports/docmedia122.tab' delimiter: "\t" enclosure: '"' header_offset: null ids: [filename] # not using most of these fields in the file import # but including because maybe needed for grouping # and migrate_lookup in the media import fields: 0: name: filename label: 'Unique filename' 1: name: title label: 'description' 2: name: doctype label: 'document type' 3: name: formflag label: 'FormYN' 4: name: newpath label: 'path' process: uid: plugin: default_value default_value: 2 uri: newpath status: plugin: default_value default_value: 1 # 1 equals 'permanent' destination: plugin: entity:file
The media file...
uuid: 1bcec3e7-0a49-4473-87a2-6dca09b91abjan-docmed id: docmedia_import label: Import media of document type migration_group: docfilesmedia source: plugin: 'csv' path: '/srv/imports/docmedia122.tab' delimiter: "\t" enclosure: '"' header_offset: null ids: [filename] fields: 0: name: filename label: 'Unique filename' 1: name: title label: 'description' 2: name: doctype label: 'document type' 3: name: formflag label: 'FormYN' 4: name: newpath label: 'path' process: name: title uid: plugin: default_value default_value: 179 field_media_document/target_id: # above is name of the file entity ref field plugin: migration_lookup migration: docfiles_import source: filename # Unclear how this works. I think it means # 'create target ids, using filename field to determine # which row to use in the csv file' field_doctype_select: plugin: entity_generate source: doctype value_key: name bundle: document_type entity_type: taxonomy_term ignore_case: true field_formcheckbox: formflag migration_dependencies: required: - docfiles_import optional:  destination: plugin: entity:media default_bundle: document
The files & media migration examples I've been able to find all have assumptions that differ from my situation (not csv, files need to be moved or created or copied, Drupal to Drupal, etc.). So I'm sure I've just made errors in adapting the guidance.
Ideas on how to do to it right or understand what's wrong? If a solution preventing the phantom file-usage items can't be found, am I probably safe removing these from the file_usage table and acting like it never happened?
I'm also seeing this error intermittently. I don't know if it's related. Since the mysql seems to work fine, I've been assuming it has something to do with using a csv rather than database source?
[error] Message: Failed to connect to your database server. The server reports the following message: /No database connection configured for source plugin variable/. * Is the database server running? * Does the database exist, and have you entered the correct database name? * Have you entered the correct username and password? * Have you entered the correct database hostname?