How to make existing public files private?

Question

Let's say I have a node type "Article" which has a field "File" which is set to handle public files. A lot of articles and files get added.

Now the specs suddenly change and the "Article" will move behind a login and therefore also the attached files must be changed to be not publicly accessible anymore.

I already found out how to change the file system for the file field from public to private pretty easily by reimporting the manually updated field.storage.node.field_file.yml config. But: this only affects new files that will be added after the switch. It doesn't work for existing files. They still are accessible publicly. Many thousand existing files would need to be manually reuploaded to also make them private.

Does anybody of you have solved that in D8 before? If so, then how?

As all I can think of right now is to programmatically query all "Article" nodes, then foreach through all nodes, get the files, temporarily store them somewhere else, empty the field out, then try to reupload the files programmatically (probably my next question) and then resave each node.

Does that sound reasonable? Or can you think of any other approach that might work here (in D8)?

Answer 1

Thanks a lot for the answers! FileField Paths definitely is the way to go.

1. Update the file field's field storage to handle private files.
2. Run FileField Paths' Retroactive update on your content type's file field.

---

Step 1. can be accomplished from within hook_update_N (if you use configuration management do that locally and then export the updated configuration and deploy the resulting *.yml):

$field_type = 'file';
$field_name = 'field_file';

if (!$field_storage_configs = \Drupal::entityManager()
  ->getStorage('field_storage_config')
  ->loadByProperties(['type' => $field_type])) {
  return;
}

foreach ($field_storage_configs as $field_storage) {
  // Note: These settings apply to the field_file field everywhere it is
  // used.
  if ($field_storage->getName() == $field_name) {
    $field_storage->setSetting('uri_scheme', 'private');
    $field_storage->enforceIsNew(FALSE);
    $field_storage->save();
  }
}

Alternatively, follow the manual config update and reimport as suggested in the question.

---

Step 2. is to go to your file field's edit screen and fire the Retroactive update upon saving the form:

Answer 2

It's quite simple, I tested this myself and it works.

1) With PHPMyAdmin or some DB managing application, you'll need to look at node__field_NAME that has your files. Record all the field_NAME_target_id numbers that belong to your content type.

2) Go to to file_managed table.

3) Export table as .sql file

4) Open .sql file with some text editing app. Rename all public:// to private:// for the ones that have the fid number that you recorded.

5) Import .sql and override table file_managed. OR you could clear the table and then import.

6) Move the files to the private folder.

• Important: make sure that it has the same parent folder structure. If it was previously under /files/2018-06/some.jpg, in private it needs to be /private/2018-06/some.jpg

  • Also, no need to move image style image files, these will get auto created. Thus, you only need to grab the original.

7) Flush all caches.

---

I'm sure you could use SQL query to do 1 to 5. Basically all you gotta do is update the specific values of the uri column in the field_managed table.

Answer 3

I did this for Drupal 7, but haven't tried it in Drupal 8.

I believe that it should work similarly since there is already a D8 release for the required module: FileField Paths. I first installed and enabled the module before following the steps below:

just in case someone is looking to do this, here are the steps:

1. backup your site and db
2. copy the table for your field to make a backup (in my case i copied: field_data_field_attachments to field_data_field_attachments_bk)
3. empty table field_data_field_attachments
4. go to drupal and the option to change the file system should be available, so change it
5. copy all data from field_data_field_attachments_bk back to field_data_field_attachments
6. go to drupal, select your content type, select your file field, in file path settings select retroactive update
7. click save and it will move all your files to the right place in file system and update your db to the new path

— A Romka

^{Full thread}

Answer 4

With my Drupal 8.6.3 the retro-active update doesn't work.

You still need to achieve step 1, to update the field storage of your field.

Then you need to update all existing files via the database with:

UPDATE file_managed SET uri = REPLACE( 
    uri,
    'public://',
    'private://'
)
WHERE filemime like 'application%';

In my case I'm updating only the files, that's why I'm using this where clause.

Don't forget to clear the cache after this SQL query.

Answer 5

I was able to resolve this without using contrib. @leymannx's answer using FileField Paths is great, but I didn't want to install a contrib module for a one-time script. If you're familiar with batch processing items in hook_post_update_NAME(), it's a piece of cake.

Step 1: Update your field to use private files. (See step 1 of @leymannx's answer)

Step 2: Make sure the new private files directory (or directories) exists (the subdirectory where you want to move your files)

Here's my example:

/**
 * Make sure the private files directory exists.
 */
function mymodule_update_9001(&$sandbox) {
  $directory = 'private://MY-SUBDIRECTORY';

  /** @var \Drupal\Core\File\FileSystemInterface $file_system */
  $file_system = \Drupal::service('file_system');

  $success = $file_system
    ->prepareDirectory($directory, FileSystemInterface::CREATE_DIRECTORY);

  if (!$success) {
    throw new UpdateException('The required directory does not exist (or exists and is not writable) or could not be created.');
  }
}

Step 3: Move existing files to the new directory.

The code to do this is straightforward for a single file:

use Drupal\file\FileInterface;

// Move file to the private files directory.
$file = loadMyFileEntitySomehow();
if ($file instanceof FileInterface) {
  // Note: $destination must be the full file uri, including file name.
  // Additionally, the directory must exist and be writable.
  $destination = 'private://MY-SUBDIRECTORY/' . $file->getFilename();
  \Drupal::service('file.repository')
    ->move($file, $destination);
}

Example post_update to update all files in a specific field on a specific node type:

use Drupal\file\FileInterface;

/**
 * Move files to the private files directory.
 */
function mymodule_post_update_make_files_private(&$sandbox) {
  $batch_size = 25;
  $entity_type = 'node';
  $bundle = 'MY_BUNDLE';

  if (!isset($sandbox['num_processed'])) {
    $sandbox['num_processed'] = 0;
  }

  if (empty($sandbox['entity_ids'])) {
    $sandbox['entity_ids'] = \Drupal::entityTypeManager()
      ->getStorage($entity_type)
      ->getQuery()
      ->condition('type', $bundle)
      ->exists('field_MY_FILE_FIELD')
      ->execute();

    if (is_array($sandbox['entity_ids'])) {
      $sandbox['total'] = count($sandbox['entity_ids']);
    }
  }

  if (!empty($sandbox['entity_ids'])) {
    $current_batch_ids = array_slice($sandbox['entity_ids'], $sandbox['num_processed'], $batch_size);
    $current_batch_entities = \Drupal::entityTypeManager()
      ->getStorage($entity_type)
      ->loadMultiple($current_batch_ids);

    foreach ($current_batch_entities as $entity) {
      // Move files to the private files directory.
      $files = $entity->field_MY_FILE_FIELD->referencedEntities();
      foreach ($files as $file) {
        if ($file instanceof FileInterface) {
          \Drupal::service('file.repository')
            ->move($file, 'private://MY-SUBDIRECTORY/' . $file->getFilename());
        }
      }

      $sandbox['num_processed']++;
    }
  }

  if (!empty($sandbox['total'])) {
    $sandbox['#finished'] = $sandbox['num_processed'] / $sandbox['total'];
  }
  else {
    $sandbox['#finished'] = 1;
  }

}

Answer 6

Thank you everybody. I combined the previous answers into two update hooks that you can mostly copy and paste. Only use this if you want to move ALL files that are currently public into private.

The only change you need to make is to the function/hook implementation names, and to the $field_names array in the first hook.

The first hook will cause config changes if you have not already set your fields to use the private directory. I would recommend making that change and exporting config locally, but including this update hook as well so that when deploying to another environment the fields are set to use the private directory before the second hook is run, which will be before config is imported (if your deploy script is in the correct order).

use Drupal\Core\File\FileSystemInterface;
use Drupal\Core\Utility\UpdateException;
use Drupal\file\FileInterface;

/**
 * Set file fields to use private files directory.
 */
function my_module_update_9007(&$sandbox) {
  $field_types = [
    'file',
    'image',
  ];
  $field_names = [
    'field_document',
    'field_media_image',
  ];

  foreach ($field_types as $field_type) {
    if (!$field_storage_configs = \Drupal::entityTypeManager()
      ->getStorage('field_storage_config')
      ->loadByProperties(['type' => $field_type])) {
      continue;
    }

    foreach ($field_storage_configs as $field_storage) {
      if (in_array($field_storage->getName(), $field_names)) {
        $field_storage->setSetting('uri_scheme', 'private');
        $field_storage->enforceIsNew(FALSE);
        $field_storage->save();
      }
    }
  }
}

/**
 * Move files from public directory to private.
 */
function my_module_update_9008(&$sandbox) {
  $batch_size = 25;

  if (!isset($sandbox['num_processed'])) {
    $sandbox['num_processed'] = 0;
  }

  if (empty($sandbox['entity_ids'])) {
    // Get an array of all file ids.
    $sandbox['entity_ids'] = \Drupal::entityQuery('file')
      ->execute();

    if (is_array($sandbox['entity_ids'])) {
      $sandbox['total'] = count($sandbox['entity_ids']);
    }
  }

  if (!empty($sandbox['entity_ids'])) {
    $current_batch_ids = array_slice($sandbox['entity_ids'], $sandbox['num_processed'], $batch_size);
    $current_batch_entities = \Drupal::entityTypeManager()
      ->getStorage('file')
      ->loadMultiple($current_batch_ids);

    foreach ($current_batch_entities as $file) {
      if ($file instanceof FileInterface) {
        $uri = $file->get('uri')->getValue()[0]['value'];
        // Only proceed if file is in public directory.
        if (str_contains($uri, 'public://')) {
          // Check that file actually exists.
          // If it doesn't there's nothing we can do.
          if (file_exists($uri)) {
            // Get the current subdirectory and recreate
            // it within the private files directory.
            $uri = str_replace('public://', '', $uri);
            $subdirectory = str_replace('/' . $file->get('filename')->getValue()[0]['value'], '', $uri);
            $subdirectory_path = 'private://' . $subdirectory;
            /** @var \Drupal\Core\File\FileSystemInterface $file_system */
            $file_system = \Drupal::service('file_system');
            $success = $file_system
              ->prepareDirectory($subdirectory_path, FileSystemInterface::CREATE_DIRECTORY);

            if (!$success) {
              throw new UpdateException('The required directory does not exist (or exists and is not writable) or could not be created.');
            }
            else {
              // Move file from public to private.
              // Note: $destination must be the full file uri,
              // including file name.
              // Additionally, the directory must exist and be writable.
              $destination = $subdirectory_path . '/' . $file->getFilename();
              \Drupal::service('file.repository')
                ->move($file, $destination);
            }
          }
        }
      }

      $sandbox['num_processed']++;
    }
  }

  if (!empty($sandbox['total'])) {
    $sandbox['#finished'] = $sandbox['num_processed'] / $sandbox['total'];
  }
  else {
    $sandbox['#finished'] = 1;
  }
}