3

Can someone help me straighten out my setup? What is needed...what isn't?

Varnish has been working well with a high hitrate average, but I am really trying to dial in my configuration. Any insight is much appreciated.

My current setup includes:

  • Varnish
  • Memcache

Modules Enabled:

  • Varnish
  • Cache Expiration
  • Purge
  • Memcache

Enabled Drupal Caching options include:

  • Cache pages for anonymous users
  • Cache blocks
  • Minimum cache lifetime: 30 mins
  • Expiration of cached pages: 6 hours
  • Compress cached pages.
  • Aggregate and compress CSS files.
  • Aggregate JavaScript files.

I've also included a snippet of my current work-in-progress 'settings.php' at the bottom of the post. I can also provide more information as needed. Purge is included in my Varnish default.vcl which is also a work in progress and at the bottom of the post.

My issues:

  • Proper 'PURGE' commands being sent to Varnish to purge only specific pages that have expired or changed.

  • What is the proper module combination to use?

The other issue I run into...

  • If I enable 'Include base URL in expires' I have to save the page twice. It seems similar to the 'Varnish / Memcache node save race condition' issue (https://www.drupal.org/node/2412517)

  • Turning off 'Purge' module allows nodes to save on first attempt with 'Include base URL in expires' enabled.

Scenario 1

When I enable 'Include base URL in expires' & Watchdog + site message in the 'Cache Expiration' module and save a node I am returned this:

Expiration was executed for the next URLs:
URL: http://www.blackhillsbadlands.com/node/2487
Wildcard: false
Expired object: node
--------
URL: http://www.blackhillsbadlands.com/business/spokane-creek-cabins-campgound-0
Wildcard: false
Expired object: node
--------
Business Spokane Creek Cabins & Campgound has been updated.

Varnish returns this when I use this ban.list command (sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban.list):

req.http.host ~ www.blackhillsbadlands.com && req.url ~ ^/node/2487$|^/business/spokane-creek-cabins-campgound-0$ 

I also see the PURGE for the page when I issue the command: varnishlog -d -c -m RxRequest:PURGE

31 ReqStart c 127.0.0.1 51589 444721271
31 RxRequest c PURGE
31 RxURL c /business/spokane-creek-cabins-campgound-0
31 RxProtocol c HTTP/1.1
31 RxHeader c Accept: */*
31 RxHeader c Host: www.blackhillsbadlands.com
31 VCL_call c recv pipe
31 VCL_call c hash
31 Hash c /business/spokane-creek-cabins-campgound-0
31 Hash c www.blackhillsbadlands.com
31 VCL_return c hash
31 VCL_call c pipe pipe
31 Backend c 36 default default
31 ReqEnd c 444721271 1447884612.290643454 1447884613.636188507 0.000025034 0.000051975 1.345493078 

Scenario 2

If I disable 'Include base URL in expires' & Watchdog + site message in the 'Cache Expiration' module and save a node I am returned this:

Expiration was executed for the next URLs:
URL:
Wildcard: false
Expired object: node
--------
URL:
Wildcard: false
Expired object: node
--------
Business Spokane Creek Cabins & Campgound has been updated. 

Varnish returns this when I use this ban.list command (sudo varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret ban.list):

req.http.host ~ www.blackhillsbadlands.com && req.url ~ ^/Array$|^/Array$

PURGE returns while using this command which seems to clear my site's entire cache: varnishlog -d -c -m RxRequest:PURGE

32 ReqStart c 208.113.205.225 34617 444723722
32 RxRequest c PURGE
32 RxURL c /
32 RxProtocol c HTTP/1.1
32 RxHeader c Accept: */*
32 VCL_call c recv pipe
32 VCL_call c hash
32 Hash c /
32 Hash c 208.113.205.225
32 VCL_return c hash
32 VCL_call c pipe pipe
32 Backend c 33 default default
32 ReqEnd c 444723722 1447885616.420059681 1447885616.420225382 0.000054836 0.000049829 0.000115871 

settings.php

// Varnish 
// Add Varnish as a cache bin.
# $conf['cache_backends'] = array('sites/all/modules/varnish/varnish.cache.inc');
 $conf['cache_backends'][] = 'sites/all/modules/varnish/varnish.cache.inc';

//If you plan to use the expire module to be selective with your cache clearing you
//should add as a new cache bin.

$conf['cache_class_external_varnish_page'] = 'VarnishCache';

// If you are not going to use the expire module you should replace the default page
// cache with varnish.
# $conf['cache_class_cache_page'] = 'VarnishCache'; 

// Bypass Drupal bootstrap for anonymous users so that Drupal sets max-age > 0
 $conf['page_cache_invoke_hooks'] = FALSE; 

# Memcache
  $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
  $conf['lock_inc'] = 'sites/all/modules/memcache/memcache-lock.inc';
  $conf['memcache_stampede_protection'] = TRUE;
  $conf['cache_default_class'] = 'MemCacheDrupal';
  $conf['memcache_servers'] = array('127.0.0.1:11211' => 'default');

  // The 'cache_form' bin must be assigned to non-volatile storage.
  $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

  // Don't bootstrap the database when serving pages from the cache.
#  $conf['page_cache_without_database'] = TRUE;
#  $conf['page_cache_invoke_hooks'] = FALSE;

default.vcl

# This is a basic VCL configuration file for varnish.  See the vcl(7) man page for details on VCL syntax and semantics.
# 
# Default backend definition.  Set this to point to your content server.
# 
 backend default {
     .host = "208.113.205.225";
     .port = "8080";
     .connect_timeout = 600s;
     .first_byte_timeout = 600s;
     .between_bytes_timeout = 600s;
 }
acl purge {
  "localhost";
  "127.0.0.1";
  "208.113.205.225";
} 
# Respond to incoming requests
 sub vcl_recv {

# Enable to disable varnish, Uncomment to enable Varnish
#    return (pipe);

# Default Varnish config START
     if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
        set req.http.X-Forwarded-For =
        req.http.X-Forwarded-For + ", " + client.ip;
    } else {
        set req.http.X-Forwarded-For = client.ip;
    }
     }
     if (req.request != "GET" &&
       req.request != "HEAD" &&
       req.request != "PUT" &&
       req.request != "POST" &&
       req.request != "TRACE" &&
       req.request != "OPTIONS" &&
       req.request != "DELETE") {
         /* Non-RFC2616 or CONNECT which is weird. */
         return (pipe);
     }

     if (req.request != "GET" && req.request != "HEAD") {
         /* We only deal with GET and HEAD by default */
         return (pass);
     }

     if (req.http.Authorization || req.http.Cookie) {
         /* Not cacheable by default */
         return (pass);
     }
#     return (lookup);
# Default Varnish config END
# Drupal START
# Use different backend for each site served through the proxy. Again, this is not needed if you're running a single site on a server.
#    if (req.http.host ~ "sitea.com$") {
#         set req.backend = sitea;
#    }
# else if (req.http.host ~ "siteb.com$") {
#        set req.backend = siteb;
#  }
#
# Use the default backend for all other requests
#    else {
#        set req.backend = default;
#  }
#  

# Use anonymous, cached pages if all backends are down.
  if(!req.backend.healthy) {
    unset req.http.Cookie;
  }

# Allow the backend to serve up stale content if it is responding slowly.
 set req.grace = 6h;

# Get rid of progress.js query params
     if (req.url ~ "^/misc/progress\.js\?[0-9]+$") {
         set req.url = "/misc/progress.js";
  } 

# Pipe these paths directly to Apache for streaming.
     if (req.url ~ "^/admin/content/backup_migrate/export") {
         return (pipe);
  }
#
#  If global redirect is on
#    if (req.url ~ "node\?page=[0-9]+$") {
#        set req.url = regsub(req.url, "node(\?page=[0-9]+$)", "\1"); return (lookup);
#  }

# Do not cache these paths.
     if (req.url ~ "^/status\.php$" ||
         req.url ~ "^/update\.php" ||
         req.url ~ "^/install\.php" ||
         req.url ~ "^/admin" ||
         req.url ~ "^/admin/.*$" ||
         req.url ~ "^/user" ||
         req.url ~ "^/user/.*$" ||
         req.url ~ "^/users/.*$" ||
         req.url ~ "^/info/.*$" ||
         req.url ~ "^/flag/.*$" ||
         req.url ~ "^.*/ajax/.*$" ||
         req.url ~ "^.*/ahah/.*$") {
         return (pass);
  }

# Disallow outside access to cron.php or install.php
#    if (req.url ~ "^/(cron|install)\.php$" && !client.ip ~ internal) {
# Either let Varnish throw the error directly,
#     error 404 "Page not found."; Or, use a custom error page that you've defined in Drupal at the path "404".
#    # set req.url = "/404";
#  }

# Always cache the following file types for all users.
     if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|css|js|html|htm)(\?[a-z0-9]+)?$") {
         unset req.http.Cookie;
  }
# Remove all cookies that Drupal doesn't need to know about. ANY remaining cookie will cause the request to pass-through to Apache. For the most part we always 
# set the NO_CACHE cookie after any POST request, disabling the Varnish cache temporarily. The session cookie allows all authenticated users to pass through as 
# long as they're logged in. @see http://drupal.stackexchange.com/questions/53467
#
# 1. Append a semi-colon to the front of the cookie string. 2. Remove all spaces that appear after semi-colons. 3. Match the cookies we want to keep, adding the 
# space we removed previously, back. (\1) is first matching group in the regsuball. 4. Remove all other cookies, identifying them by the fact that they have no 
# space after the preceding semi-colon. 5. Remove all spaces and semi-colons from the beginning and end of the cookie string.
     if (req.http.Cookie) {
         set req.http.Cookie = ";" + req.http.Cookie;
         set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
         set req.http.Cookie = regsuball(req.http.Cookie, ";(S{1,2}ESS[a-z0-9]+|NO_CACHE)=", "; \1=");
         set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
         set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");

# Remove the "has_js" cookie
         set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");
# Remove the "Drupal.toolbar.collapsed" cookie
         set req.http.Cookie = regsuball(req.http.Cookie, "Drupal.toolbar.collapsed=[^;]+(; )?", "");
# Remove AdminToolbar cookie for drupal6
         set req.http.Cookie = regsuball(req.http.Cookie, "DrupalAdminToolbar=[^;]+(; )?", "");
# Remove any Google Analytics based cookies
         set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
# Remove the Quant Capital cookies (added by some plugin, all __qca)
         set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");
# If there are no remaining cookies, remove the cookie header. If there aren't any cookie headers, Varnish's default behavior will be to cache the page.
     if (req.http.Cookie == "") {
         unset req.http.Cookie;
    }

# If there are any cookies left (a session or NO_CACHE cookie), do not cache the page; pass it on to Apache directly.
     else {
      return (pass);
    }
  }
# Check the incoming request type is "PURGE", not "GET" or "POST"
     if (req.request == "PURGE") {
# Check if the ip coresponds with the acl purge
     if (!client.ip ~ purge) {
# Return error code 405 (Forbidden) when not
     error 405 "Not allowed.";
    }
     return (lookup);
  }
     return (lookup);
 }
 sub vcl_hit {

     if (req.request == "PURGE") {
         purge;
     error 200 "Purged.";
  }

     return (deliver);
 }

 sub vcl_miss {

     if (req.request == "PURGE") {
         purge;
     error 200 "Purged.";
  }

     return (fetch);
 }

 sub vcl_fetch {
     if (beresp.ttl <= 0s ||
         beresp.http.Set-Cookie ||
         beresp.http.Vary == "*") {
        /*
         * Mark as "Hit-For-Pass" for the next 2 minutes
         */
        set beresp.ttl = 120 s;
        return (hit_for_pass);
     }
     return (deliver);
 }

 sub vcl_deliver {
     return (deliver);
 }

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.