Migrating multiple values into a single file field in Drupal 7 with Migrate 2.2

Question

I have a file field that can have multiple downloadable files for an image node type. These images will have several versions (psd, jpg, eps) that will be downloadable. I would like to be able to append multiple values to this field. How would I do that when they are coming from multiple source columns (i.e. eps_filename, jpg_filename, psd_filename)?

I found related documentation here: http://drupal.org/node/1012810 But that documentation is for taxonomy terms where the details of the value can simply be a string with a separator. For this I am using MigrateFileFieldHandler::arguments(), so I can't just give it a separator.

Further what I would like to do is also be able to come back later and append more files to the migrated nodes. This is because the larger files will take longer to get on the server, so their smaller counterparts (jpg, png, etc) will be loaded first and then adding the eps files later. Now I already figured out how to update the previously migrated nodes using systemOfRecord. But I am not sure how to append to a multi-value field.

I am using Drupal 7 and the Migrate module version 2.2. Here is some sample code I am using for migrating the data initially into the file field.

$jpg_arguments = MigrateFileFieldHandler::arguments(NULL,
  'file_link', FILE_EXISTS_RENAME, 'en', array('source_field' => 'jpg_name'),
  array('source_field' => 'jpg_filename'), array('source_field' => 'jpg_filename'));

$this->addFieldMapping('field_file', 'jpg_uri')
     ->arguments($jpg_arguments);

I suspect the trick is using the prepareRow() function. But so far I have not been able to get that to execute yet.

Any ideas? I really appreciate any help.

Thanks!

Answer 1

I think what you need is prepareRow. Here is code from the migrate baseball example where multiple fields are merged into a single array field.

 function prepareRow($row) {
    // Collect all the batters into one multi-value field.
    for ($i=1; $i <= 9; $i++ ) {
      $key = "visiting_batter_$i";
      $visiting_batters[] = $row->$key;
      $key = "home_batter_$i";
      $home_batters[] = $row->$key;
    }
    $row->visiting_batters = implode(',', $visiting_batters);
    $row->home_batters = implode(',', $home_batters);
    $row->title = "$row->home_team vs. $row->visiting_team. " . gmdate('M d, Y', strtotime($row->start_date));
  }

(from the file migrate_example_baseball.migrate.inc)

However I have been having mixed results with prepareRow in my own code, so while this looks straight forward I can't guarantee that it will work.

Answer 2

hmmrr. It might be easier not to do this in 1 pass ... but a 2 step migration. Create your new parent record in 1 migration task, then back-fill in your N file_fields with the new node or entity id as a second migration task.

furthermore, while you've looked into System of Record -- my understanding of this feature is it only works for DB schemas which are 1:1 between your old DB and the new DB schema. Eg, a multiple value field entry cannot be inspected because could be N-fields in a row in an old DB and for instance N-rows crossreferenced to a parent row id in the new DB.

I have used the "Query in Prepare" method, I tried the group_concat version too .... so that's my 2 cents on the matter.

Answer 3

Actually I figured this out last month, but just got back to commenting here. The solution was far more complex than what anyone has suggested. I actually had to extend the MigrateDestinationNode class and create my own import function that is a modified version of the one used in that class. I will paste the source code below.

Essentially what I had to do was modify the import where it tries to do an update on the existing node when the System of Record is set to destination to check for the field type that I am saving the file to and merge the file field data of the file that is already on the node with the one I am adding.

I also had to make sure that I used this new extended class as the destination and set the System of Record to Migration::DESTINATION.

Here is the code:

<?php
class ImageAssetNodeAddEPSMigration extends BasicClientMigration {
  public function __construct() {
    parent::__construct();

    //This allows us to update the existing record.
    $this->systemOfRecord = Migration::DESTINATION;

    $this->description = t('Image Asset Files - Adding the EPS files');
    $this->dependencies = array('ImageAssetNode');

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'destid1' => array(
          'type' => 'int',
          'not null' => TRUE,
          'description' => 'Node ID.',
          'alias' => 'an',
        )
      ),
      MigrateDestinationNode::getKeySchema()
    );

    // We get the data from the database for this.
    $query = db_select('migrate_map_imageassetnode', 'an')
        ->fields('an', array('destid1'));
    $query->innerJoin('migrate_client_image_asset_node', 'ta', 'an.sourceid1 = ta.fcid');
    $query->fields('ta', array('eps_psd_filename', 'eps_psd_name', 'eps_psd_uri', 'fcid') );

    // Passing the cache_counts option means the source count (shown in
    // drush migrate-status) will be cached - this can be very handy when
    // dealing with a slow source database.
    $this->source = new MigrateSourceSQL($query, array(), NULL,
      array('cache_counts' => TRUE));

    // Set up our destination - nodes of type migrate_example_beer
    $this->destination = new MigrateDestinationNodeExtended('media_images');

    /*
      Assign mappings TO destination fields FROM source fields. To discover
      the names used in these calls, use the drush commands
      drush migrate-fields-destination ImageAssetNode
      drush migrate-fields-source ImageAssetNode
    */

    // Mapped fields
    $this->addFieldMapping('nid', 'destid1')
         ->description(t('The original node id.'));

    // Copy an image file, write DB record to files table, and save in Field storage.
    // Note we specify the source query fields that will map to the file alt/title/description
    // values.

    $eps_arguments = MigrateFileFieldHandler::arguments(NULL,
      'file_link', FILE_EXISTS_RENAME, 'en', array('source_field' => 'eps_psd_name'),
      array('source_field' => 'eps_psd_filename'), array('source_field' => 'eps_psd_filename'), NULL, TRUE);

    $this->addFieldMapping('field_file', 'eps_psd_uri')
         ->arguments($eps_arguments);

    // Add fields that are mapped via argument
    #$this->addFieldMapping(NULL, 'eps_psd_uri');
    $this->addFieldMapping(NULL, 'eps_psd_filename');
    $this->addFieldMapping(NULL, 'eps_psd_name');

    // No unmapped source fields
    $this->addUnmigratedSources( array('fcid', 'jpg_uri', 'jpg_filename', 'jpg_name', 'category_tid') );

    // Unmapped destination fields
    $this->addUnmigratedDestinations( array('is_new', 'name', 'created', 'changed',
      'status', 'promote', 'revision', 'language', 'sticky', 'uid', 'revision_uid',
      'title', 'body', 'field_image', 'field_keywords', 'field_download_count',
      'field_images_category', 'field_media_status', 'field_media_audience',
      'field_roles_download', 'field_roles_share', 'field_roles_view', 'path', 'comment',
      'bookmarks', 'cart', 'download', 'pathauto') );
  }
  /*
  public function prepare(&$node, &$row) {

  }*/

  public function prepareRow($current_row) {
    //If the file does not exist then skip it.
    if (!file_exists(drupal_realpath($current_row->eps_psd_uri))) {
      return FALSE;
    }

    $result = db_query("SELECT ff.* FROM field_data_field_file ff INNER JOIN migrate_map_imageassetnode ian ON ff.entity_id = ian.destid1 WHERE sourceid1 = :sid", array(':sid' => $current_row->fcid));

    //If there is more then one file field entry attached to the node then skip.
    if ($result->rowCount() > 1) {
      return FALSE;
    }
  }
}

class MigrateDestinationNodeExtended extends MigrateDestinationNode {
  /**
   * Basic initialization
   *
   * @param string $bundle
   *  A.k.a. the content type (page, article, etc.) of the node.
   * @param array $options
   *  Options applied to nodes.
   */
  public function __construct($bundle, array $options = array()) {
    parent::__construct($bundle, $options);
  }

  /**
   * Import a single node.
   *
   * @param $node
   *  Node object to build. Prefilled with any fields mapped in the Migration.
   * @param $row
   *  Raw source data object - passed through to prepare/complete handlers.
   * @return array
   *  Array of key fields (nid only in this case) of the node that was saved if
   *  successful. FALSE on failure.
   */
  public function import(stdClass $node, stdClass $row) {
    // Updating previously-migrated content?
    $migration = Migration::currentMigration();

    if (isset($row->migrate_map_destid1)) {
      // Make sure is_new is off
      $node->is_new = FALSE;

      if (isset($node->nid)) {
        if ($node->nid != $row->migrate_map_destid1) {
          throw new MigrateException(t("Incoming nid !nid and map destination nid !destid1 don't match",
            array('!nid' => $node->nid, '!destid1' => $row->migrate_map_destid1)));
        }
      }
      else {
        $node->nid = $row->migrate_map_destid1;
      }

      // Get the existing vid, tnid so updates don't generate notices
      $values = db_select('node', 'n')
                   ->fields('n', array('vid', 'tnid'))
                   ->condition('nid', $node->nid)
                   ->execute()
                   ->fetchAssoc();

      $node->vid = $values['vid'];
      $node->tnid = $values['tnid'];
    }

    if ($migration->getSystemOfRecord() == Migration::DESTINATION) {
      if (!isset($node->nid)) {
        throw new MigrateException(t('System-of-record is DESTINATION, but no destination nid provided'));
      }

      $old_node = node_load($node->nid);

      if (!isset($node->created) && isset($old_node->created)) {
        $node->created = $old_node->created;
      }

      if (!isset($node->vid) && isset($old_node->vid)) {
        $node->vid = $old_node->vid;
      }

      if (!isset($node->status) && isset($old_node->status)) {
        $node->status = $old_node->status;
      }

      if (!isset($node->uid) && isset($old_node->uid)) {
        $node->uid = $old_node->uid;
      }
    }

    // Set some required properties.
    // Set type before invoking prepare handlers - they may take type-dependent actions.
    $node->type = $this->bundle;

    if ($migration->getSystemOfRecord() == Migration::SOURCE) {
      if (!isset($node->language)) {
        $node->language = $this->language;
      }

      // Apply defaults, allow standard node prepare hooks to fire.
      // node_object_prepare() will blow these away, so save them here and
      // stuff them in later if need be.
      if (isset($node->created)) {
        $created = MigrationBase::timestamp($node->created);
      }
      else {
        // To keep node_object_prepare() from choking
        $node->created = REQUEST_TIME;
      }

      if (isset($node->changed)) {
        $changed = MigrationBase::timestamp($node->changed);
      }

      if (isset($node->uid)) {
        $uid = $node->uid;
      }

      node_object_prepare($node);

      if (isset($created)) {
        $node->created = $created;
      }

      // No point to resetting $node->changed here, node_save() will overwrite it
      if (isset($uid)) {
        $node->uid = $uid;
      }
    }

    // Invoke migration prepare handlers
    $this->prepare($node, $row);

    if (!isset($node->revision)) {
      $node->revision = 0; // Saves disk space and writes. Can be overridden.
    }

    // Trying to update an existing node
    if ($migration->getSystemOfRecord() == Migration::DESTINATION) {
      // Incoming data overrides existing data, so only copy non-existent fields
      if ( !isset($old_node) || ( isset($old_node) && (empty($old_node) || !is_array($old_node)) )) {
        $return = FALSE;
      }
      else {
        foreach ($old_node as $field => $value) {
          // An explicit NULL in the source data means to wipe to old     value (i.e.,
          // don't copy it over from $old_node)
          if (property_exists($node, $field) && $node->$field === NULL) {
            // Ignore this field
          }
          elseif ($field == 'field_file') {
            $file_field = $node->$field;
            $lang_files = array();

            // We can import this, got to do it on a per language basis
            foreach ($value AS $lang => $files) {
              if (isset($file_field[$lang])) {
                $new_node_files = $file_field[$lang];
              }
              else {
                $new_node_files = array();
              }

              $lang_files[$lang] = array_merge($new_node_files, $files);
            }

            $node->$field = $lang_files;
          }
          elseif (!isset($node->$field)) {
            $node->$field = $old_node->$field;
          }
        }
      }

    }

    if (isset($node->nid) && !(isset($node->is_new) && $node->is_new)) {
      $updating = TRUE;
    }
    else {
      $updating = FALSE;
    }

    migrate_instrument_start('node_save');
    node_save($node);
    migrate_instrument_stop('node_save');

    if (isset($node->nid)) {
      if ($updating) {
        $this->numUpdated++;
      }
      else {
        $this->numCreated++;
      }

      // Unfortunately, http://drupal.org/node/722688 was not accepted, so fix
      // the changed timestamp
      if (isset($changed)) {
        db_update('node')
          ->fields(array('changed' => $changed))
          ->condition('nid', $node->nid)
          ->execute();

        $node->changed = $changed;
      }

      // Potentially fix uid and timestamp in node_revisions.
      $query = db_update('node_revision')
                 ->condition('vid', $node->vid);

      if (isset($changed)) {
        $fields['timestamp'] = $changed;
      }

      $revision_uid = isset($node->revision_uid) ? $node->revision_uid : $node->uid;

      if ($revision_uid != $GLOBALS['user']->uid) {
        $fields['uid'] = $revision_uid;
      }

      if (!empty($fields)) {
        // We actually have something to update.
        $query->fields($fields);
        $query->execute();

        if (isset($changed)) {
          $node->timestamp = $changed;
        }
      }

      $return = array($node->nid);
    }
    else {
      $return = FALSE;
    }

    $this->complete($node, $row);
    return $return;
  }
}
?>

Pastebin Link as well: http://pastebin.com/ykzgVvCF