How to determine reason for unprocessed files in Drupal to Drupal migration

Question

Finally ran my first file migration from Drupal 6 to Drupal 7 using UI.

• 3100 Items
• 3036 Processed
• 62 Unprocessed
• 2498 messages

I am simply trying to figure out what those 62 files are, and why they did not process.

I looked at the "messages" link for the file migration task. They all say something like:

2701 Informational Deleted documents from index with query id:"pyli4t/file/3782" OR sm_parent_document_id:"pyl4t/file/3782" Don't know what to do with this, or how to query for unprocessed files.

I looked at the migrate_map_43b7eadb1file table for the migration task, but don't see anything that indicates what files were not processed.

I looked at the migrate_message_43b7eadb1file table and was able to locate the two files that failed using this SQL statement: SELECT * FROM migrate_message_43b7eadb1file WHERE level != 4 But, nothing about unprocessed files.

That's all I can think to do. Any suggestions?

Answer 1

It looks like this poster received an answer from the Drupal-to-Drupal data migration module maintainer over at drupal.org, so for convenience I will post it here:

Source:Issue#2476077: How to determine reason for unprocessed files

Side issue first - that informational message is coming from some contrib module (something search-related, perhaps) and not related to migration per-se. Grep your codebase for "Deleted documents from index" to see what it might be.

So, the unprocessed count is precisely the count of source records, minus the number of records in the map table (so, you will never find an "unprocessed" record in the map table. All processed records are recorded in the map table with an appropriate status.

The first question is, are there really unprocessed records, or is the source count off? The most common cause of "unprocessed" records being reported is a malformed source query that has more than one row with the same primary key (fid in this case) - this can happen when joining to another table that has multiple rows per "base" row. Elsewhere you've been speaking about extending migrate_d2d classes and modifying their queries - could you have done that for the file query, and joined to another table?

The next step is to try to identify specific unprocessed files. If your D6 and D7 databases are on the same MySQL server and accessible with the same MySQL user, you can do something like:

select d6.fid 
from d6.files f
left join d7.migrate_map_myfilemigration map on f.fid=map.sourceid1
where map.sourceid1 is null

If the dbs are not on the same server, you could try dumping the D6 files table (or just the fids) and loading it into a temp table on the D7 server.

Answer 2

In our case the reason for the unprocessed items was that we were returning FALSE in our source plugin's prepareRow method, after calling parent::prepareRow.

The base method includes the following code to explicitly mark the row as ignored:

$this->idMap->saveIdMapping($row, [], MigrateIdMapInterface::STATUS_IGNORED);