Migration of multi-value pictures to user fields

Question

I need to migrate from a WordPress site to a Drupal 7 site where users could upload their own images (like multiple user pictures). I have a CSV file with the User's ID and a serialized array containing the filenames of uploaded pictures (no, really).

Users have a multi-value image field 'field_user_pictures'.

After deserializing the array, drupal complains with the error "ltrim() expects parameter 1 to be string, array given". (Presumably because MigrateDestinationFile doesn't handle arrays very well.) Then I tried with just using the first image of the array, and that actually works. What doesn't work, however, is to get drupal to link this sole image to the 'field_user_pictures' user field.

I have been trying to get to something like this:

 class PictureMigration extends Migration
 {
    public function __construct ($arguments)
        {
        parent::__construct($arguments);

        $source = 'user_pictures.csv';

        $columns = array(
            1 => array('source_userid', ''),
            3 => array('source_serialized', ''),
        );

        $source_key = array('source_userid' => array('type' => 'int', 'not null' => TRUE),);

        $this->source = new MigrateSourceCSV($source, $columns);
        $this->destination = new MigrateDestinationFile('image', 'MigrateFileUri');
        $this->map = new MigrateSQLMap($this->machineName, $source_key, MigrateDestinationFile::getKeySchema());

        $this->addFieldMapping('source_dir')->defaultValue($this->remote_dir);
        $this->addFieldMapping('destination_dir')->defaultValue('public://');
        $this->addFieldMapping('value', 'source_serialized');
      }

  public function prepareRow($row) {
     // snipped
     $row->source_serialized = $this->deserializePhotos($row->source_serialized); // returns array of filenames
    }   
}

public class UserMigration extends Migration {
  public class __construct ($arguments) {
    // snipped...
      $this->addFieldMapping('field_user_pictures')->sourceMigration('PictureMigration');
  }
}

The first problem is that MigrateDestinationFile doesn't seem to like multi-value values; the second problem is that even when I only use the first filename of the array, it still doesn't get linked to the 'field_user_pictures' field...

See also: Migration: Combine Profile2 and file migration -- almost exactly the same issue, but with no valid answer as well.

Answer 1

Something is really odd about your code $this->destination is a MigrateDestinationFile yet $this->map is trying to track a MigrateDestinationNode .... I've never done anything that way.

I would do the following:

1. Import Users from Wordpress to Drupal.
2. Use your CSV file of legacy UIDs and picture files to run an update of the (newly) created Drupal users and insert the images as needed.

I'm doing this now as we speak on a Document content type of which there are 120k entries, but only 4k have movie files associated with them so I feel a node revision is a better use the time to import stuff.

This is an example from some work in progress and will not run as-is but it should be a concrete example:

class PrezDocumentMediaMigration extends PrezMigration {

  public function __construct($arguments) {
    parent::__construct($arguments);

    $document_fields = array(
      // ... fields ...
    );
    $this->description = 'Update created Documents with audio and video from legacy DB. There are only ~5k of entries.';

    $query = Database::getConnection('default', 'legacy_migration')
      ->select('papers', 'p')
      ->fields('p', $document_fields)
      ->condition('av', 0, '<>');
    // Additional pseudo fields to move data from legacy derived data to drupal.
    $query->addExpression('NULL', 'local_video');
    $query->addExpression('NULL', 'remote_video');
    $query->addExpression('NULL', 'local_audio');
    $query->addExpression('NULL', 'remote_audio');

    $options = array('track_changes' => 1);

    $count_query = Database::getConnection('default', 'legacy_migration')
      ->select('papers', 'p')
      ->condition('av', 0, '<>');
    $count_query->addExpression('COUNT(pid)', 'cnt');

    // We are updating existing content in Drupal.
    $this->systemOfRecord = Migration::DESTINATION;

    $this->source = new MigrateSourceSQL($query, $document_fields, $count_query, $options);
    $this->destination = new MigrateDestinationNode('documents');
    $this->map = new MigrateSQLMap($this->machineName,
                                   array(
                                     'pid' => array(
                                       'type'     => 'int',
                                       'unsigned' => FALSE,
                                       'not null' => TRUE,
                                       'description' => 'Unique content reference legacy document',
                                       'alias' => 'doc',
                                     )
                                   ),
                                   MigrateDestinationNode::getKeySchema()
    );

    // Migration mappings, the documentation for these mappings can be seen at:
    // https://www.drupal.org/node/1133448

    $migrator_user = user_load_by_name('PrezMigrator');

    // Some default values for sanity.
    $this->addFieldMapping('uid')
      ->defaultValue($migrator_user->uid);
    $this->addFieldMapping('revision')
      ->defaultValue(1);
    $this->addFieldMapping('log')
      ->defaultValue('inserted related media from legacy DB on ' . date("l jS of F g:i A.", time()));
    $this->addFieldMapping('promote')->defaultValue(0);
    $this->addFieldMapping('sticky')->defaultValue(0);
    $this->addFieldMapping('revision_uid')->defaultValue($migrator_user->uid);

    // What legacy content are we talking about ?
    // also we want this legacy ID to actually lookup through 2 source migrations.
    $this->addFieldMapping('nid', 'id')->sourceMigration(array('PrezDietNonPresidents','PrezDietPresidents'));

    // Map media files these are multiple value fields. They can accept a PHP array of values.
    $this->addFieldMapping('field_docs_media_videos_local', 'local_video');
    $this->addFieldMapping('field_docs_media_videos_remote', 'remote_video');
    $this->addFieldMapping('field_docs_media_audo_local', 'local_audio');
    $this->addFieldMapping('field_docs_media_audio_remote', 'remote_audio');

    // Each of the above media fields will need the following additional subfields
    // Only 1 is provided as an example to keep this example shorter:
    $this->addFieldMapping('field_docs_media_videos_local:file_class')
      ->defaultValue('MigrateFileUri');
    $this->addFieldMapping('field_docs_media_videos_local:language')
      ->defaultValue(LANGUAGE_NONE);
    $this->addFieldMapping('field_docs_media_videos_local:preserve_files')
      ->defaultValue(FALSE);
    $this->addFieldMapping('field_docs_media_videos_local:file_replace')
      ->defaultValue(FILE_EXISTS_REPLACE);
    $this->addFieldMapping('field_docs_media_videos_local:source_dir')
      ->defaultValue('/var/www/2015/presidency/drupal7-project/build/legacy/htdocs/www/media/');
    $this->addFieldMapping('field_docs_media_videos_local:urlencode')
      ->defaultValue(FALSE);
    $this->addFieldMapping('field_docs_media_videos_local:alt')->defaultValue('');
    $this->addFieldMapping('field_docs_media_videos_local:title')
      ->defaultValue('');


    // List fields not be migrated to destination.
    $this->addUnmigratedDestinations(
      array(
        // ... massive list of fields.
      ));

  }

  public function prepareRow($row) {
    // Always include this fragment at the beginning of every prepareRow()
    // implementation, so parent classes can ignore rows.
    if (parent::prepareRow($row) === FALSE) {
      return FALSE;
    }

    $youtube_format = 3;
    $quicktime_format = 2;
    $audio_format = 1;

    switch ($row->av) {
      case $youtube_format:
        $row->remote_video = $this-> find_remote_video($row->medialink);
        break;
      case $quicktime_format:
        $row->local_video = $this->find_local_video($row->medialink);      
        break;
      case $audio_format:
        $row->local_audio = $this->find_local_audio($row->av, $row->medialink);        
      default:
        break;
    }
    return TRUE;
  }

  // http://stackoverflow.com/questions/5830387/how-to-find-all-youtube-video-ids-in-a-string-using-a-regex
  protected function find_remote_video($link) {
      $separator = '***';
    $text = preg_replace('~
        # Match non-linked youtube URL in the wild. (Rev:20130823)
        https?://         # Required scheme. Either http or https.
        (?:[0-9A-Z-]+\.)? # Optional subdomain.
        (?:               # Group host alternatives.
          youtu\.be/      # Either youtu.be,
        | youtube         # or youtube.com or
          (?:-nocookie)?  # youtube-nocookie.com
          \.com           # followed by
          \S*             # Allow anything up to VIDEO_ID,
          [^\w\s-]       # but char before ID is non-ID char.
        )                 # End host alternatives.
        ([\w-]{11})      # $1: VIDEO_ID is exactly 11 chars.
        (?=[^\w-]|$)     # Assert next char is non-ID or EOS.
        (?!               # Assert URL is not pre-linked.
          [?=&+%\w.-]*    # Allow URL (query) remainder.
          (?:             # Group pre-linked alternatives.
            [\'"][^<>]*>  # Either inside a start tag,
          | </a>          # or inside <a> element text contents.
          )               # End recognized pre-linked alts.
        )                 # End negative lookahead assertion.
        [?=&+%\w.-]*        # Consume any URL (query) remainder.
        ~ix',
                         '//www.youtube.com/embed/$1' . $separator,
                         $link);
    return explode($separator, $text);
  }

  // other "find" media methods that return filesystem paths and stuff depending on where the media
  // existed in the legacy system.
}