Here is the solution that I came up with.
TLDR: Drush command to compare source data vs existing products, and activate or deactivate products accordingly.
Use case / goal
I am focusing on a specific use case, to make this easier to explain. Something similar could be done for other use cases.
Commerce products are imported with a migration. Both source and destination have one numeric primary key. The migrate map is MigrateSQLMap, so there is a migrate_map_* table in the database with destid1 and sourceid1.
Goal: Change the "status" property of commerce products which are no longer in the source data.
A migration run can be incomplete. In such a case, we do not want remaining products to be deactivated.
Updating an entity (commerce product) in Drupal takes time. We do not want to update every entity just to change a timestamp.
We have to assume that the entire list of products can be quite big, so it can be problematic to have them all in memory at once. However, in this solution, I assume that having the ids (source id mapped to destination id) in memory is still acceptable.
Solution: Custom drush command
Create a custom drush command that does the following:
Query the database, join the
commerce_product AS cp table with the
migrate_map_MYMIGRATION AS mm table. For each row we want
cp.status. Load the complete list into an array as
$maps[$row->status][$row->sourceid1] = $row->destid1.
Assign the two parts of the array to new variables
$inactiveIdsMap = $maps; and
$activeIdsMap = $maps;.
Create two additional associative arrays, each with format
$[$sourceid] = $destid:
$idsToActivateMap = ;. Start value: empty array.
$idsToDeactivateMap = $activeIdsMap. Start value: All ids for active products.
You may want to
unset($activeIdsMap); to free memory. (We could have reused this variable instead of declaring a new one, but I like having variable names that are accurate throughout a variables's life.)
Iterate through the source data. You can get the migrate source from the migration. It is a bit tricky but it can be made to work.
For each source row:
- If the id exists in
$inactiveIdsMap, add the source + dest id to the
- In any case, unset this entry from the
Now deactivate all to-be-deactivated products, and activate all to-be-activated products.
You ideally do this in chunks, so e.g. you load 20 products at once with commerce_product_load_multiple(), set each one's status, and save them individually with commerce_product_save().
This drush command can be executed after the migration.
You can measure memory use with
memory_get_peak_usage(). If you use
memory_get_usage(), you need to call it in the place where you expect memory usage to be highest. Memory pretty print: http://codeaid.net/php/convert-size-in-bytes-to-a-human-readable-format-(php)
The way to check for the impact of this specific operation is to run the command with a subset of data. E.g. put a
->range() on the select query for loading the migrate map. This should make all arrays smaller.
Another solution that I found tempting at first was to maintain a "lastseen" timestamp in a separate table.
However, this would mean either having to update each product on every migration run, or at least reindex the search index each time, because we would want to filter by "lastseen".
With the solution presented here, in the average case, only few products have to be updated.
It turned out that the memory impact was very manageable.