6

Open the logs of a dozen Drupal sites and you will see some patterns for non-Drupal packages in the http requests that received a 404 response. Requests like /postnuke/article.php, /exchange/logon.asp, /awstats.pl, /wp-content/, /mailman/, /phpBB/page_header.php, etc, etc.

Nearly 100% of these are bots scanning for exploits. I know there are several modules to block these requests based on IP, http header, or some other key piece of data.

I've been mulling over the idea of writing a module that approaches this differently.

Instead of trying to block the request, just cache the response. For sites using a reverse-proxy layer cache like Varnish, set the max-age for a ridiculously long time (1 year). The module would simply generate menu entries for common packages that a Drupal site is very unlikely to have installed. I would include the option to exclude a specific package so if you really did want to run Drupal and phpBB in the same web root, you could... but may require you to install https://www.drupal.org/project/bad_judgement to do that :)

I realize that this type of configuration can also be done at the .htaccess level, but maintaining that for all of the packages bots are trying to exploit is beyond the skill set of many people.

Does something like this already exist?

Am I missing some obvious reason this wouldn't work. It seems like this would improve performance of a site by simply never letting a 2nd request hit the php/mysql level for another year (or until you cleared the Varnish cache)?

closed as off-topic by kiamlaluno Oct 19 '15 at 8:19

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking to recommend or find a book, tool, module, theme, distribution, tutorial, or other off-site resource are off-topic for Drupal Answers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – kiamlaluno
If this question can be reworded to fit the rules in the help center, please edit the question.

4

You can alter the fast404 code inside of setting.php to handle this. It will handle *.asp & *.pl; changing the regex so it handles paths like

/postnuke/article.php  
/wp-content/  
/mailman/  
/phpBB/page_header.php  

Can be done by replacing this

$conf['404_fast_paths'] = '/\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

with this

$conf['404_fast_paths'] = '/^(postnuke|wp-content|mailman|phpBB)|\.(?:txt|png|gif|jpe?g|css|js|ico|swf|flv|cgi|bat|pl|dll|exe|asp)$/i';

All i did was add ^(postnuke|wp-content|mailman|phpBB)| to the start of the string, right after the first /. This means that if any path starts with any of these strings we'll skip booting drupal and 404 right here taking very little server resources.

To enable fast404

Go to the sites/default directory. Open up to the setting.php file. Around line 500 uncomment this drupal_fast_404(); by removing #

Source: https://docs.google.com/presentation/d/1AgnurTWsWdMAFjfFmSU7mk2zfJu8jG_KmVdZ7-o6Pok/edit#slide=id.g385c64257_0178

If you have dblog enabled you can run this query to find what paths are the worst offenders

SELECT
  COUNT(*) AS count,
  watchdog.*
FROM watchdog AS watchdog
WHERE type = 'page not found'
AND message NOT LIKE 'sites/all/%'
AND message NOT LIKE 'sites/default/%'
AND message NOT LIKE '%/styles/%'
GROUP BY message
ORDER BY
  count DESC,
  wid ASC
  • That works, but adding regex to the settings.php is very similar to the .htaccess approach. I could also stop these requests at the load balancer level so they never even get to the http server... assuming I have a load balancer and know how to configure it. I'm thinking about a solution for a site builder level user who is hosting with a service that is already providing varnish. This may also work with Boost. – kreynen Mar 25 '15 at 16:02
  • Any executed code after settings.php will start the database connection & start to eat up server resources. You could make hits to bot pages cacheable but unless you're using varnish then it won't achieve your original goal of "never letting a 2nd request hit the php/mysql level". Having pages like /exchange/logon.asp in the drupal cache as a 404 would be better than nothing; but the drupal cache doesn't allow for non 200s. The best you can do while keeping it easy is to use hook_boot; adding a patch to the drupal.org/project/fast_404 module is the route I would go in your case. – mikeytown2 Mar 25 '15 at 18:22
  • Sorry. Maybe I'm missing the obvious, but if I cache the first request for /exchange/logon.asp with a max-age of 1 year, why wouldn't the 2nd request be served a response from varnish? In addition 3rd, 4th, and all other bot requests that URL for the next year? I understand that I will see more of a hit on the first request than the fast_404 or .htaccess approaches, but currently these sites let all of these requests get to the php/mysql level. – kreynen Mar 25 '15 at 20:19
  • 1
    drupal core doesn't cache anything other than a 200. Varnish can be configured to cache things besides a 200, not sure if it has been configured to do so in your case. garron.me/en/bits/avoid-varnish-cache-404-error-page.html Odds are if you're using some form of hosted varnish, they will limit how long a 404 is cached for. – mikeytown2 Mar 25 '15 at 20:24
  • Thanks! The issue a cache timeout for 404s that might override max-age is exactly the information I needed! – kreynen Mar 25 '15 at 21:42
0

It would be preferable to catch these request at the outermost part of your server, Varnish. By adding a line like the following to your VCL:

     if (req.url ~ "/wp-admin/" || req.url ~ "/fckeditor*" || req.url ~ "/wp-login.php" || req.url ~ "/pma/scripts/" || req.url ~ "/myadmin/scripts/" || req.url ~ "/MyAdmin/scripts/" || req.url ~ "/wp-content/plugins/" || req.url ~ "/blog/wp-i*") {
        return (synth(403, "Forbidden"));
  }

you catch the troublesome requests and do what you want with them, give the a forbidden or a redirect somewhere else.

I think(...) it would be possible to create a VMOD that would be able to read a list of 'banned URLs' which was kept updated by a Drupal module.

Not the answer you're looking for? Browse other questions tagged or ask your own question.