Private files not found 404 error

Question

My installation is Drupal 7 + Nginx + Varnish cache

My private files path: sites/default/files/private/demo.img URL in browser: system/files/demo.img

When I try to open private file: http://domian.com/system/files/demo.img server throws 404 not found error. So I check all the file system config and directory permissions, everything seems to be fine.

Now I tried to debug Drupal core files, what I have understood is,

1. System.module has defined url "system/files" with callback function "file_download" using hook_menu()

2. function file_download() is defined in 'includes/file.inc' core file and it is not invoked when I try to access the file: http://domian.com/system/files/demo.img

So my private file request is not handled by Drupal, which is wrong? In my local setup though, which has Apache instead of Nginx, private files are normally accessed from Drupal as is expected.

So is my Nginx conf is wrong? Pasting my nginx site configuration:

server {
    server_name www.demo.com;
    root /var/www/demo/; ## <-- Your only path reference.

listen 127.0.0.1:8080;

location = /check.php {
     access_log off;
     log_not_found off;
  }


location /nginx_status {
       stub_status on;
       access_log   off;
       allow 127.0.0.1;
       deny all;
}


location ~ ^/(status|ping)$ {
        include fastcgi_params;
        fastcgi_pass 127.0.0.1:9000;
        fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
        allow 127.0.0.1;
        #allow stats_collector.localdomain;
        #allow watchdog.localdomain;
        deny all;
}


# Enable compression, this will help if you have for instance advagg‎ module
# by serving Gzip versions of the files.
#gzip_static on;

location = /favicon.ico {
        log_not_found off;
        access_log off;
}

location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
}

# This matters if you use drush prior to 5.x
# After 5.x backups are stored outside the Drupal install.
#location = /backup {
#        deny all;
#}

# Very rarely should these ever be accessed outside of your lan
location ~* \.(txt|log)$ {
        #allow 192.168.0.0/16;
        deny all;
}

location ~ \..*/.*\.php$ {
        return 403;
}

# No no for private
location ~ ^/sites/.*/private/ {
        internal;
}

# Block access to "hidden" files and directories whose names begin with a
# period. This includes directories used by version control systems such
# as Subversion or Git to store control files.
location ~ (^|/)\. {
        return 403;
}

location / {
        # This is cool because no php is touched for static content
        try_files $uri @rewrite;

 location ~* ^(?:.+\.(?:htaccess|make|txt|engine|inc|info|install|module|profile|po|pot|sh|.*sql|test|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template)$ {
         return 404;
 }

rewrite ^/(.*)\.css /$1\.css\.gz;
rewrite ^/(.*)\.js /$1\.js\.gz;


}


location @rewrite {
        # You have 2 options here
        # For D7 and above:
        # Clean URLs are handled in drupal_environment_initialize().
        rewrite ^ /index.php;
        # For Drupal 6 and bwlow:
        # Some modules enforce no slash (/) at the end of the URL
        # Else this rewrite block wouldn't be needed (GlobalRedirect)
        #rewrite ^/(.*)$ /index.php?q=$1;
}

location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        #NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
        include fastcgi_params;
        fastcgi_read_timeout 120;
        fastcgi_param SCRIPT_FILENAME $request_filename;
        fastcgi_intercept_errors on;
        fastcgi_pass phpbackend;
}

# Fighting with Styles? This little gem is amazing.
# This is for D6
#location ~ ^/sites/.*/files/imagecache/ {
# This is for D7 and D8
location ~ ^/sites/.*/files/styles/ {
        try_files $uri @rewrite;
}

location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        expires max;
        log_not_found off;
}

}

Latest development: The private file which I am trying to view is webform submission and While I try to view file, the function webform_submission_access() see logged in user as anonymous.

Answer 1

Ok, I got this issue fixed.

The Vanish installed on server was causing the issue more specifically the following code.

# Always cache the following file types for all users.

    if (req.url ~ "(?i)\.(pdf|woff|eot|svg|ttf|png|gif|jpeg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
      unset req.http.Cookie;
      # Stop processing further for those file types. This way, we ensure to have
      # X-Varnish-Cache HIT for those requests.
      return (lookup);
    }

You can either remove the file extensions from above code or comment the entire block. This should solve the issue.

Answer 2

The accepted answer will solve the issue but will also hit the site's performance. Instead, consider not caching these specific requests.

For example, in the beginning of Varnish vcl_recv:

  # Do not cache requests to private files.
  if (req.url ~ "^/system/files/.*$") {
    return (pass);
  }

Yo may combine this with other request paths:

  # Do not cache these paths.
  if (req.url ~ "^/status\.php$" ||
      req.url ~ "^/update\.php$" ||
      req.url ~ "^/admin$" ||
      req.url ~ "^/admin/.*$" ||
      req.url ~ "^/flag/.*$" ||
      req.url ~ "^.*/ajax/.*$" ||
      req.url ~ "^.*/ahah/.*$" ||
      req.url ~ "^/system/files/.*$") {
    return (pass);
  }

Reference: https://www.drupal.org/node/2375961

Answer 3

As well as this, you can sometimes have some issues serving images from the private folder if you're following Perusio's config and your private file folder is within the folder sites/default/files.

The thing to do is alter in your nginx config from

~*\.(jpg|jpeg|png|gif|css|js|ico)$ to /sites/default/files/~*\.(jpg|jpeg|png|gif|css|js|ico)$