0

We are having a high traffic Drupal 6.x site (25,000) uniques per day, mostly landing pages.

Everythings works OK, with great performance, until we try to reset the boost cache.

Once we reset it, Drupal tries to regenerate HTML from PHP and MYSQL, and we reach a performance crisis where the site crawls and becomes unusable due to slow performance.

Is there a better solution? I imagine there could be a script which clears cache slowly, resets files one by one, wait(10 seconds), wget the url, and then go to the next url.

More info added (edits):

  • The site has about 20,000 nodes.
  • We have two Linode 4096 machines: 4 GB RAM, 8 CPU (4x priority), 192 GB Storage, 8 TB Transfer each. Only one of them stores MySQL.
  • The reason we do manual purge is because when Boost purges pages by itself, we can get to a point where a lot of pages are not in the cache. This can cause a big performance hit. When the server is too busy, PHP returns WSOD and then Boost caches it! So we get a page, in cache that is actually WSOD!!! Since this is a critical problem, we decided to affectively disable Boost from purging cache on it own and do it On Demand.
  • Why clear the boost cache? The CSS and JS files are global. If we change the CSS, or add some JS, then we "drush cc css-js" to load the new changes. Doing that, the CSS/JS files are deleted and we are left with cached HTML pointing to non-existing CSS/JS files. So we get "broken" pages. The solution is to refresh the Boost cache.

Will the advagg module help?

Any other solutions?

Pleae advise.

  • 1
    There is no ready-made script like that, not in Boost itself. You pretty much know how to write it, so feel free and invited to post script as an answer. Last but not least - WHY?! Boost can invalidate only portions of it's cache, like ones that are affected by the node edit. Why to purge it all? – Mołot Sep 30 '13 at 18:12
1

Here is the script that does the job:

<?php

/* function to list all relevant HTML files in a given directory */
function ListIn($dir, $prefix = '') {
  $dir = rtrim($dir, '\\/');
  $result = array();

    foreach (scandir($dir) as $f) {
      if ($f !== '.' and $f !== '..') {
        if (is_dir("$dir/$f")) {
          $result = array_merge($result, ListIn("$dir/$f", "$prefix$f/"));
        } else {
          if (stripos($f,".gz") === FALSE) {
            $result[] = $prefix.$f;
          }
        }
      }
    }

  return $result;
}

/* main function */
function go() {
  $isToNotify = false;
  // get all relevant HTML files, recursively
  $files = ListIn('/var/www/drupal_site/cache/normal/my.domain.com','');
  foreach($files as $file) {
    // save filename for later
    $filename = str_replace("_.html","",$file);
    // normalize path to file
    $file = "/var/www/drupal_site/cache/normal/my.domain.com/" . $file;
    echo "\n************************************\n\nChecking $file" . "...\n\n";
    // get HTML from file into string variable
    $html = file_get_contents($file);
    // create DOM object for easy parsing
    $dom = new DOMDocument;
    $dom->loadHTML($html);
    // get all <link> tags
    $links = $dom->getElementsByTagName("link");
    foreach($links as $link) {
      $css = $link->getAttribute('href');
      $css = str_replace("http://my.domain.com","",$css);
      // find only the relevant tags with the CSS file
      if (stripos($css,"css_") !== FALSE) {
        // normalize path to CSS file
        $path = "/var/www/drupal_site" .$css;
        // check if CSS exists
        if (file_exists($path)) {
          echo "The file $path exists\n";
          // no need to do anything else
        }
        else {
          echo "The file $path does not exist\n";
          // set notification ON
          $isToNotify = true;
          // delete HTML with links to not existing CSS file
          unlink("/var/www/drupal_site/cache/normal/my.domain.com/$filename"."_.html");
          unlink("/var/www/drupal_site/cache/normal/my.domain.com/$filename"."_.html.gz");
          // GET the HTML from scratch so Boost will re-cache it
          echo file_get_contents("http://my.domain.com/$filename");
          echo "Adding log entry...\n";
          // log activity
          $content = "\n\n".(date("Y-m-d H:i:s"))."\n    Refreshed HTML: my.domain.com/".$filename."\n    CSS file which was not found: ".$css;
          file_put_contents("/root/scripts/logs/monitorCSS.log",$content,FILE_APPEND);
          echo "Going to sleep for a while ZzZzZz...\n";
          // sleep for a while to avoid bombardment of the server by this script
          sleep(20); // in seconds
          // go to next file - no need to refresh same file more than once (if more than one CSS file was not-found in HTML)
          break;
        }
      }
    }
  }
  if ($isToNotify) {
    // send email notification if needed (we found some missing CSS files)
    mail('my@email.address', 'Monitor CSS Notification', wordwrap("Some missing CSS files discovered - see log file for more info...", 70));
  }
}

// call main function
go();
?>
-1

You mention the amount of visitors but not the number of nodes. That is likely more important here, since Boost will want to rebuild it's cache.

That said, why are you trying to flush the cache often? With boost, you typically don't need to do an entire wipe of the page cache unless you're actually changing things often. Rather, you can set a minimum cache lifetime and then just let it do it's ting.

Unless of course, you have a use case I'm not picturing.

Lastly, what size Linode do you have? It's possible that you're hitting a hardware limit for the amount of content and traffic you have.

  • There are not much nodes - about 20,000. – Druvision Oct 1 '13 at 14:12
  • Boost rebuilds cache on access, so no matter how many nodes one have, if only X nodes are accessed per hour after purge, Boost will generate the same amount of resource usage, regenerating X node pages only. Unless OP knows it's different in his case. – Mołot Oct 1 '13 at 14:19
  • I've detailed the use-case above. This is not a hardware limit. – Druvision Oct 1 '13 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.