Compression in Drupal 7

Question

I am running Drupal (latest version 7.22 ) on Apache 2.2 and I have also installed Varnish (module and proxy). In Apache, I have disabled the mod_deflate module. Reading around the web, it seems that the options on the performance page in Drupal (aggregate css,js) shouldn't compress the css and js files. However, browsing my site and checking the http headers, I see that I am receiving "Content-Encoding: gzip". Looking at the (default) .htaccess, I see that there are some rewrite rules in order to server gziped files to those clients that can read them. So, I guess this is where the "Content-Encoding: gzip" headers come from. Also, when I enable compression of cached pages on the performance settings page in drupal, I find that even the html of the page I am requesting (as anonymous user only) is returned compressed. My questions on the above are:

1) Do the options "Aggregate and compress CSS files." and "Aggregate JavaScript files." actually compress the aggregated files or is it just that .htaccess makes them seem like gziped? Using chrome developer tools, the files seem compressed. Checking them with the developer toolbar addon in firefox, however, (Information -> View document size) they are not reported as compressed (while the html file is always being reported as such, even when I access it as authenticated user!)

2) If they are indeed gziped, where does the compression take place? Can I control the level of compression, etc somehow?

3) I am always receiving X-Drupal-Cache "MISS". I don't care much since I have Varnish installed. Yet, since I am not hitting the cache, how come I am returned a compressed html file (that is not in the cache)?

4) Is there any chance that the Varnish module messes anything up? I am accessing the website with https so as to be sure that Varnish is bypassed, in any case.

5) If I enable mod_deflate, should I leave that part of .htaccess intact? I understand that mod_deflate doesn't allow for precompression, but what if the compression is better?

6) Also, if drupal compresses on its own the css, js and html files, what is the point of enabling mod_deflate for these files?

Here is the relevant part of my .htaccess file for reference:

  <IfModule mod_headers.c>
    # Serve gzip compressed CSS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

    # Serve gzip compressed JS files if they exist and the client accepts gzip.
    RewriteCond %{HTTP:Accept-encoding} gzip
    RewriteCond %{REQUEST_FILENAME}\.gz -s
    RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

    # Serve correct content types, and prevent mod_deflate double gzip.
    RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
    RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

    <FilesMatch "(\.js\.gz|\.css\.gz)$">
      # Serve correct encoding type.
      Header set Content-Encoding gzip
      # Force proxies to cache gzipped & non-gzipped css/js files separately.
      Header append Vary Accept-Encoding
    </FilesMatch>
  </IfModule>

Answer 1

A start:

1) Yes.

Note: If I really want to know what is going over the line, I use fiddler (Windows).

2) See function drupal_build_css_cache() in common.inc (line 3603). You cannot influence the level but you can influence whether it compresses or not with variable 'css_gzip_compression', see default.settings.php line 432.

BTW: You do not want to influence the compression level: for compress once, use many cases, maximum compression is always the best. Transfer time and decompressing time at the client is linear with compressed document size. So a 1-time slightly increased performance hit will save a lot of network activity and client side processing in the future.

3) -

4) -

5) Compression will be better as it will be done only once and it will be done better versus mod_deflate that will compress on each request and therefore will use a lower compression level. I don't know much about mod_deflate, but I guess that if there is already a Content-Encoding header that tells the content is compressed, mod_deflate should not touch it. But I have had cases where I got a backup file that was compressed twice, so mod_deflate might not be that smart and should explicitly be told to do so.

6) There is no point, but there might be for other requests, especially dynamic ones, i.e. non cacheable pages. Note that only aggregated css and js and cached pages (variable 'page_compression') are compressed by Drupal. Non cacheable pages are not compressed by Drupal.

Answer 2

To sum up what I have found so far and fietserwin's answer.

1) Those options do compress. No matter what you have read elsewhere. I went through some bugs in the drupal bug tracker and there are a couple bugs (eg this discussion for d8) that discuss actually separating this behavior with different options in the Performance settings in drupal. Also, you can check it with Chromium devtools or Firefox Network panel (not through the size column, but by checking the response headers for Content-Encoding: gzip) and toggling the available options for css/js aggregation in drupal. The developer toolbar seems broken or I cannot understand what it does.

You can have aggregation but no compression by setting the css_gzip_compression and js_gzip_compression variables to FALSE (see fietserwin).

2) The compression happens in php code. For more detail, look fietswin's answer. Judging by his answer on the compression level, I guess that php (at least for drupal) compresses at the maximum level of compression. Note that compressing in php and not relying on the web server is common for many other php applications, eg owncloud, flyspray, etc.

3) This is a bit weird and still haven't figured out why it happens. I think it has to do with the varnish module. I am not saying varnish itself, since I am running varnish 2 with no compression support). It only happens when I access the site through varnish (ie, when I access it with https, which bypasses varnish, the page is not cached and not gzipped). And if I have the "Compress cached pages" option enabled. So, my guess is that the varnish module completely takes over the caching mechanism of drupal but still uses these options to set the behavior (compression,lifetime,etc) of the varnish cache.

Since I couldn't find any more info on this and I have mod_deflate set to gzip all html files, I disabled this option to prevent the cached pages from being double compressed. Maybe there is a check or sth, as with css/js, to prevent this but I haven't found one.

4) obsolete question

5) Compression should happen in drupal, not in apache. There, it only happens once (in a while). So the .htaccess excerpt above should be left intact. What it does is:

RewriteCond %{HTTP:Accept-encoding} gzip #if the client can handle compression
RewriteCond %{REQUEST_FILENAME}\.gz -s #and if the (aggregated css file already exists
RewriteRule ^(.*)\.css $1\.css\.gz [QSA] #serve the compressed file instead

# The same for js files
RewriteCond %{HTTP:Accept-encoding} gzip
RewriteCond %{REQUEST_FILENAME}\.gz -s
RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

# Do no actual rewriting but set the no-gzip apache variable to inform apache not to compress the already compressed files.
RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]

So this part should be left intact to safely take advantage of the whole mechanism. The compression, as said above, probably is the best possible.

6) To add to fietserwin's answer, there might be other apps running on the server as well. All in all, just enable compression for css and js in apache as well, and drupal's htaccess will prevent double compression.