Special code for .htaccess and robots.txt files

Question

My client was noticing that his site was extremely slow yesterday morning so he contacted his WiredTree host, who suggested that we add the following code in the .htaccess file:

Options +FollowSymlinks 
RewriteBase / 
SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^DotBot" bad_user
SetEnvIfNoCase User-Agent "^AhrefsBot" bad_user
Deny from env=bad_user

Options +FollowSymlinks is already in the file and RewriteBase / is currently commented out and I'm not sure if I should uncomment it, not knowing what affect that would have. Then I would add the 3 set commands and the deny command below it. Is it OK to uncomment this line and add the extra commands?

Answer 1

It looks like the most important bit they want you to put in .htaccess is this:

SetEnvIfNoCase User-Agent "^SemrushBot" bad_user
SetEnvIfNoCase User-Agent "^DotBot" bad_user
SetEnvIfNoCase User-Agent "^AhrefsBot" bad_user
Deny from env=bad_user

They should probably go inside the <IfModule mod_rewrite.c> block in your .htaccess file, just after RewriteBase / (whether it's commented out or not). This RewriteBase / line shouldn't cause any problems, if your drupal site is not in a subdirectory (e.g. your index.php file shouldn't be in a sub-folder of your webroot like http://domain.com/drupal/index.php)

The Options +FollowSymLinks shouldn't need to be included twice if it's already there.