3

I'm migrating pictures into Drupal (9) media entities (image). System is using AWS S3 for local storage. I have 2 migrations:

First migrates external images into drupal's files. It copies files from S3 bucket where export is to S3 bucket used as drupal local storage and registers the files.

Second is parsing same data source but creating image media entities from those previously migrated image files.

With second migration, for most of the files it works well, but for some I get message like:

[error]  File 'private://images/2.0_138.jpg' could not be copied because a file by that name already exists in the destination directory ('').

And migration stops. If I roll back that row and try to migrate it

Stack trace when that happens looks like:

#0 /var/www/web/core/lib/Drupal/Core/File/FileSystem.php(298): Drupal\Core\File\FileSystem->prepareDestination('...', false, 2)
#1 /var/www/web/modules/contrib/s3fs/src/S3fsFileService.php(244): Drupal\Core\File\FileSystem->copy('...', false, 2)
#2 /var/www/web/modules/contrib/imagemagick/src/EventSubscriber/ImagemagickEventSubscriber.php(129): Drupal\s3fs\S3fsFileService->copy('...', '...', 2)
#3 /var/www/web/modules/contrib/imagemagick/src/EventSubscriber/ImagemagickEventSubscriber.php(217): Drupal\imagemagick\EventSubscriber\ImagemagickEventSubscriber->doEnsureSourceLocalPath(Object(Drupal\imagemagick\ImagemagickExecArguments))
#4 [internal function]: Drupal\imagemagick\EventSubscriber\ImagemagickEventSubscriber->ensureSourceLocalPath(Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#5 /var/www/web/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(142): call_user_func(Array, Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#6 /var/www/web/modules/contrib/imagemagick/src/Plugin/FileMetadata/ImagemagickIdentify.php(242): Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object(Drupal\imagemagick\Event\ImagemagickExecutionEvent), '...')
#7 /var/www/web/modules/contrib/imagemagick/src/Plugin/FileMetadata/ImagemagickIdentify.php(114): Drupal\imagemagick\Plugin\FileMetadata\ImagemagickIdentify->identify()
#8 /var/www/web/modules/contrib/file_mdm/src/Plugin/FileMetadata/FileMetadataPluginBase.php(292): Drupal\imagemagick\Plugin\FileMetadata\ImagemagickIdentify->doGetMetadataFromFile()
#9 /var/www/web/modules/contrib/file_mdm/src/Plugin/FileMetadata/FileMetadataPluginBase.php(389): Drupal\file_mdm\Plugin\FileMetadata\FileMetadataPluginBase->loadMetadataFromFile()
#10 /var/www/web/modules/contrib/file_mdm/src/FileMetadata.php(187): Drupal\file_mdm\Plugin\FileMetadata\FileMetadataPluginBase->getMetadata(NULL)
#11 /var/www/web/modules/contrib/imagemagick/src/Plugin/ImageToolkit/ImagemagickToolkit.php(743): Drupal\file_mdm\FileMetadata->getMetadata('...')
#12 /var/www/web/core/lib/Drupal/Core/Image/Image.php(53): Drupal\imagemagick\Plugin\ImageToolkit\ImagemagickToolkit->parseFile()
#13 /var/www/web/core/lib/Drupal/Core/Image/ImageFactory.php(85): Drupal\Core\Image\Image->__construct(Object(Drupal\imagemagick\Plugin\ImageToolkit\ImagemagickToolkit), '...')
#14 /var/www/web/core/modules/image/src/Plugin/Field/FieldType/ImageItem.php(316): Drupal\Core\Image\ImageFactory->get('...')
#15 /var/www/web/core/lib/Drupal/Core/Field/FieldItemList.php(233): Drupal\image\Plugin\Field\FieldType\ImageItem->preSave()
#16 /var/www/web/core/lib/Drupal/Core/Field/FieldItemList.php(191): Drupal\Core\Field\FieldItemList->delegateMethod('...')
#17 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(941): Drupal\Core\Field\FieldItemList->preSave()
#18 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(891): Drupal\Core\Entity\ContentEntityStorageBase->invokeFieldMethod('...', Object(Drupal\media\Entity\Media))
#19 /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php(563): Drupal\Core\Entity\ContentEntityStorageBase->invokeHook('...', Object(Drupal\media\Entity\Media))
#20 /var/www/web/core/lib/Drupal/Core/Entity/ContentEntityStorageBase.php(756): Drupal\Core\Entity\EntityStorageBase->doPreSave(Object(Drupal\media\Entity\Media))
#21 /var/www/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php(517): Drupal\Core\Entity\ContentEntityStorageBase->doPreSave(Object(Drupal\media\Entity\Media))
#22 /var/www/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php(804): Drupal\Core\Entity\EntityStorageBase->save(Object(Drupal\media\Entity\Media))
#23 /var/www/web/core/modules/media/src/MediaStorage.php(29): Drupal\Core\Entity\Sql\SqlContentEntityStorage->save(Object(Drupal\media\Entity\Media))
#24 /var/www/web/core/lib/Drupal/Core/Entity/EntityBase.php(339): Drupal\media\MediaStorage->save(Object(Drupal\media\Entity\Media))
#25 /var/www/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(240): Drupal\Core\Entity\EntityBase->save()
#26 /var/www/web/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php(179): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->save(Object(Drupal\media\Entity\Media), Array)
#27 /var/www/web/core/modules/migrate/src/MigrateExecutable.php(248): Drupal\migrate\Plugin\migrate\destination\EntityContentBase->import(Object(Drupal\migrate\Row), Array)
#28 /var/www/vendor/drush/drush/includes/drush.inc(122): Drupal\migrate\MigrateExecutable->import()
#29 /var/www/vendor/drush/drush/includes/drush.inc(113): drush_call_user_func_array(Array, Array)
#30 /var/www/vendor/drush/drush/src/Drupal/Commands/core/MigrateRunnerCommands.php(455): drush_op(Array)
#31 [internal function]: Drush\Drupal\Commands\core\MigrateRunnerCommands->executeMigration(Object(Drupal\migrate\Plugin\Migration), '...', Array)
#32 /var/www/vendor/drush/drush/src/Drupal/Commands/core/MigrateRunnerCommands.php(405): array_walk(Array, Array, Array)
#33 [internal function]: Drush\Drupal\Commands\core\MigrateRunnerCommands->import('...', Array)
#34 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(257): call_user_func_array(Array, Array)
#35 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(212): Consolidation\AnnotatedCommand\CommandProcessor->runCommandCallback(Array, Object(Consolidation\AnnotatedCommand\CommandData))
#36 /var/www/vendor/consolidation/annotated-command/src/CommandProcessor.php(176): Consolidation\AnnotatedCommand\CommandProcessor->validateRunAndAlter(Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#37 /var/www/vendor/consolidation/annotated-command/src/AnnotatedCommand.php(350): Consolidation\AnnotatedCommand\CommandProcessor->process(Object(Symfony\Component\Console\Output\ConsoleOutput), Array, Array, Object(Consolidation\AnnotatedCommand\CommandData))
#38 /var/www/vendor/symfony/console/Command/Command.php(255): Consolidation\AnnotatedCommand\AnnotatedCommand->execute(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#39 /var/www/vendor/symfony/console/Application.php(1039): Symfony\Component\Console\Command\Command->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#40 /var/www/vendor/symfony/console/Application.php(275): Symfony\Component\Console\Application->doRunCommand(Object(Consolidation\AnnotatedCommand\AnnotatedCommand), Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#41 /var/www/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#42 /var/www/vendor/drush/drush/src/Runtime/Runtime.php(121): Symfony\Component\Console\Application->run(Object(Drush\Symfony\DrushArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#43 /var/www/vendor/drush/drush/src/Runtime/Runtime.php(51): Drush\Runtime\Runtime->doRun(Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#44 /var/www/vendor/drush/drush/drush.php(72): Drush\Runtime\Runtime->run(Array)
#45 /var/www/vendor/drush/drush/drush(4): require('...')
#46 /var/www/vendor/bin/drush(120): include('...')

So what happens (how I understood stack trace) is that when image media is created some meta data are checked (image dimensions) and ImageMagick is used for that. It can not work with remote files so it's doEnsureSourceLocalPath is called. That method figures out that image is not local and tries to copy it. It calls copy from S3fsFileService class, but it assumes that file is not on S3 so calls copy from FileSystem. That one tries to copy file but I guess again S3 wrapper class is called (?) to copy file over it self and that fails.

Now I don't understand why that happens when majority of images are processed well and image media entities are created. I don't see anything specific about those which fail.

Any idea why this happens?

Update: Place where exception is thrown (web/core/lib/Drupal/Core/File/FileSystem.php prepareDestination() method) looks like:

// Determine whether we can perform this operation based on overwrite rules.
$destination = $this->getDestinationFilename($destination, $replace);
echo "Destination: $destination\n";
if ($destination === FALSE) {
  $this->logger->error("File '%original_source' could not be copied because a file by that name already exists in the destination directory ('%destination').", [
    '%original_source' => $original_source,
    '%destination' => $destination,
  ]);
  throw new FileExistsException("File '$original_source' could not be copied because a file by that name already exists in the destination directory ('$destination').");
}

I added that echo to see what proper destination looks like and it's something like: temporary://imagemagick_C3os7P.jpg But for rows which are failing it's empty string.

Improve this question
4
  • I'm not sure even if ImageMagick is failing copying file to temp dir or from temp to some permanent place.
    – MilanG
    May 12 at 10:32
  • Can you paste your migration config? Looks like you need to tell to replace your file if it exists. If you are using the file_copy plugin, you need to specify exists: replace.
    – milkovsky
    May 19 at 10:51
  • Migration that reports that error is not copying any file. Image media entity generation triggers that copy operation.
    – MilanG
    May 20 at 20:03
  • 1
    According to the error it is clear, that there are multiple parties involved: imagemagick, s3, media image entity, file permissions, etc. The problem could be in any of them. 1) The migration config yaml content would still be useful to investigate the problem. 2) I would also strongly recommend to use xdebug for troubleshooting of any problems in migration.
    – milkovsky
    May 22 at 8:22

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.