How do I change the type of a field attached to a content entity?

Question

So I've been banging my head against this for a few hours, and I'm probably going to give up and just uninstall/reinstall the module (and update the View that relies on the old entity_reference field and thus makes that fail).

A bit about my use case: I originally created some entity_reference fields pointing at files. I then learned that the File module will only automatically track file usage on file fields. I want to convert my fields to file fields and preserve the data. This seems impossible, however.

Consider this update code:

/**
 * Use file fields instead of entity_reference fields for referring to files.
 */
function fillpdf_update_8103() {
  $edum = \Drupal::entityDefinitionUpdateManager();
  $em = \Drupal::entityManager();

  $form_file_def = BaseFieldDefinition::create('file')
    ->setLabel(t('The associated managed file.'))
    ->setDescription(t('The associated managed file.'))
    ->setName('file')
    ->setProvider('fillpdf_form')
    ->setTargetBundle(NULL)
    ->setTargetEntityTypeId('fillpdf_form');

  $fc_file_def = BaseFieldDefinition::create('file')
    ->setLabel(t('The associated managed file.'))
    ->setDescription(t('The associated managed file.'))
    ->setName('file')
    ->setProvider('fillpdf_file_context')
    ->setTargetBundle(NULL)
    ->setTargetEntityTypeId('fillpdf_file_context');

  // Save existing fillpdf_form data.
  $form_files = [];
  $forms = $em->getStorage('fillpdf_form')->loadMultiple();
  foreach ($forms as $form) {
    $form_files[$form->id()] = $form->file->target_id;
    $form->file = NULL;
    $form->save();
  }

  // Save existing fillpdf_file_context data.
  $fc_files = [];
  $fcs = $em->getStorage('fillpdf_file_context')->loadMultiple();
  foreach ($fcs as $fc) {
    $fc_files[$fc->id()] = $fc->file->target_id;
    $fc->file = NULL;
    $fc->save();
  }

  // Now install the new field definitions.
  $edum->updateFieldStorageDefinition($form_file_def);
  $edum->updateFieldStorageDefinition($fc_file_def);

  foreach ($form_files as $entity_id => $fillpdf_form_file) {
    /** @var ContentEntityInterface $entity */
    $entity = $em->getStorage('fillpdf_form')->load($entity_id);
    $entity->file->target_id = $fillpdf_form_file;
    $entity->save();

    // We want postSave() hooks to run so that file usage gets recorded.
    // @see ContentEntityStorageBase::invokeFieldMethod().
    $result = [];
    // So $update = FALSE gets passed to the handler.
    $args = [FALSE];
    $method = 'postSave';
    foreach (array_keys($entity->getTranslationLanguages()) as $langcode) {
      $translation = $entity->getTranslation($langcode);
      // For non translatable fields, there is only one field object instance
      // across all translations and it has as parent entity the entity in the
      // default entity translation. Therefore field methods on non translatable
      // fields should be invoked only on the default entity translation.
      $fields = $translation->isDefaultTranslation() ? $translation->getFields() : $translation->getTranslatableFields();
      foreach ($fields as $name => $items) {
        if ($name === 'file') {
          // call_user_func_array() is way slower than a direct call so we avoid
          // using it if have no parameters.
          $result[$langcode][$name] = $args ? call_user_func_array([
            $items,
            $method,
          ], $args) : $items->{$method}();
        }
      }
    }
  }

  foreach ($fc_files as $entity_id => $ffcf) {
    /** @var ContentEntityInterface $entity */
    $entity = $em->getStorage('fillpdf_file_context')->load($entity_id);
    $entity->file->target_id = $ffcf;
    $entity->save();

    // We want postSave() hooks to run so that file usage gets recorded.
    // @see ContentEntityStorageBase::invokeFieldMethod().
    $result = [];
    // So $update = FALSE gets passed to the handler.
    $args = [FALSE];
    $method = 'postSave';
    foreach (array_keys($entity->getTranslationLanguages()) as $langcode) {
      $translation = $entity->getTranslation($langcode);
      // For non translatable fields, there is only one field object instance
      // across all translations and it has as parent entity the entity in the
      // default entity translation. Therefore field methods on non translatable
      // fields should be invoked only on the default entity translation.
      $fields = $translation->isDefaultTranslation() ? $translation->getFields() : $translation->getTranslatableFields();
      foreach ($fields as $name => $items) {
        if ($name === 'file') {
          // call_user_func_array() is way slower than a direct call so we avoid
          // using it if have no parameters.
          $result[$langcode][$name] = $args ? call_user_func_array([
            $items,
            $method,
          ], $args) : $items->{$method}();
        }
      }
    }
  }
}

Don't worry too much about the hack to run $field->postSave(). That actually works fine.

The main issue is that if I update the FieldStorageDefinition for file BEFORE I run the update, it fails because then the calls to loadMultiple() try to use the new schema.

If I leave the code alone (a.k.a. don't change the schema), it runs alright, but it re-saves the data using the old field type, and thus the postSave hooks don't actually run...at least not the one I want, FileFieldItemList::postSave().

I don't find it surprising that the change record uses the database to save the data to be re-inserted; that's probably deliberate, since otherwise you get the aforementioned schema errors. But doing that is in no way portable. I mean, what if they aren't using SQL storage?

The ideal solution in my head would be to be able to get the storage manager to retrieve the entity based on the old definition before I update the schema and the new definition after updating it. However, the logic that retrieves the old schema is tied up in protected functions, e.g. SqlContentEntityStorage::doLoadMultiple().

Supporting information: this is the error I get when I try to run the update after having updated the schemas in-module:

fillpdf module

Update #8103

- Failed: Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'base.file__target_id' in 'field list': SELECT base.fid AS fid, base.uuid AS uuid, base.file__target_id AS file__target_id, base.file__display AS file__display, base.file__description AS file__description, base.admin_title AS admin_title, base.title AS title, base.default_entity_type AS default_entity_type, base.default_entity_id AS default_entity_id, base.destination_path AS destination_path, base.scheme AS scheme, base.destination_redirect AS destination_redirect FROM {fillpdf_forms} base; Array ( ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->getFromStorage() (line 429 of /var/www/site/docroot/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

Is my only option to create a new field, move the data, and just deprecate the old field? I don't want to do that because I don't think I'll ever be able to remove it while schema storage keeps working like this, lest a future update fail when trying to load entities.

Answer 1

Talking with chx and catch on IRC, I learned that hook_update_N() is an SQL-only world, pretty much precisely for the reason I encountered. I couldn't find any issues clearly stating this, but hook_update_N() itself does say this:

Be careful about API functions and especially CRUD operations that you use in your update function. If they invoke hooks or use services, they may not behave as expected, and it may actually not be appropriate to use the normal API functions that invoke all the hooks, use the database schema, and/or use services in an update function -- you may need to switch to using a more direct method (database query, etc.).
In particular, loading, saving, or performing any other CRUD operation on an entity is never safe to do (they always involve hooks and services).

So, with that in mind, my finished update hook became the following.

<?php
/**
 * Use file fields instead of entity_reference fields for referring to files.
 */
function fillpdf_update_8103() {
  $edum = \Drupal::entityDefinitionUpdateManager();
  $em = \Drupal::entityManager();
  $db = \Drupal::database();

  $form_file_def = BaseFieldDefinition::create('file')
    ->setLabel(t('The associated managed file.'))
    ->setDescription(t('The associated managed file.'))
    ->setName('file')
    ->setProvider('fillpdf_form')
    ->setTargetBundle(NULL)
    ->setTargetEntityTypeId('fillpdf_form');

  $fc_file_def = BaseFieldDefinition::create('file')
    ->setLabel(t('The associated managed file.'))
    ->setDescription(t('The associated managed file.'))
    ->setName('file')
    ->setProvider('fillpdf_file_context')
    ->setTargetBundle(NULL)
    ->setTargetEntityTypeId('fillpdf_file_context');

  // Save existing data.
  $form_files = $db->select('fillpdf_forms', 'ff')
    ->fields('ff', ['fid', 'file'])
    ->execute()
    ->fetchAllKeyed();

  $fc_files = $db->select('fillpdf_file_context', 'fc')
    ->fields('fc', ['id', 'file'])
    ->execute()
    ->fetchAllKeyed();

  // Remove data from the storage.
  $db->update('fillpdf_forms')
    ->fields(['file' => NULL])
    ->execute();

  $db->update('fillpdf_file_context')
    ->fields(['file' => NULL])
    ->execute();

  // Now install the new field definitions.
  $edum->updateFieldStorageDefinition($form_file_def);
  $edum->updateFieldStorageDefinition($fc_file_def);

  foreach ($form_files as $entity_id => $fillpdf_form_file) {
    /** @var ContentEntityInterface $entity */
    $entity = $em->getStorage('fillpdf_form')->load($entity_id);
    $entity->file->target_id = $fillpdf_form_file;
    $entity->save();
  }

  foreach ($fc_files as $entity_id => $ffcf) {
    /** @var ContentEntityInterface $entity */
    $entity = $em->getStorage('fillpdf_file_context')->load($entity_id);
    $entity->file->target_id = $ffcf;
    $entity->save();
  }
}
?>

Note that I'm kind of ignoring the thing about not using CRUD, and I use it after the schema has been updated. I didn't want to deal with the kind of INSERT query I saw on https://www.drupal.org/node/2554097#change-field-schema, and it worked just fine. Your mileage may vary there if dealing with more complex entities, but it worked great for me.