0

My client was noticing that his site was extremely slow yesterday morning so he contacted his WiredTree host, who suggested that we add the following code in the .htaccess file:

Options +FollowSymlinks 
RewriteBase / 
SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^DotBot" bad_user
SetEnvIfNoCase User-Agent "^AhrefsBot" bad_user
Deny from env=bad_user

Options +FollowSymlinks is already in the file and RewriteBase / is currently commented out and I'm not sure if I should uncomment it, not knowing what affect that would have. Then I would add the 3 set commands and the deny command below it. Is it OK to uncomment this line and add the extra commands?

  • Welcome to Drupal Answers! The only part that is on-topic for us is the part about the .htaccess file, since Drupal comes with one, and changes to it could interfere on Drupal. Explaining how to use the robots.txt file is off-topic since Disallow: / has the same effect whenever you are using Drupal or not. – kiamlaluno Jun 15 '16 at 10:22
2

It looks like the most important bit they want you to put in .htaccess is this:

SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^DotBot" bad_user
SetEnvIfNoCase User-Agent "^AhrefsBot" bad_user
Deny from env=bad_user

They should probably go inside the <IfModule mod_rewrite.c> block in your .htaccess file, just after RewriteBase / (whether it's commented out or not). This RewriteBase / line shouldn't cause any problems, if your drupal site is not in a subdirectory (e.g. your index.php file shouldn't be in a sub-folder of your webroot like http://domain.com/drupal/index.php)

The Options +FollowSymLinks shouldn't need to be included twice if it's already there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.