1

I have just put a Drupal website online. I registered it in Google WebMaster's tool and it sent me back a Severe health issues are found on your site message related to a robot.txt file.

The content of this file is:

#
# robots.txt
#
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
#
# This file will be ignored unless it is at the root of your host:
# Used:    http://example.com/robots.txt
# Ignored: http://example.com/site/robots.txt
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/wc/robots.html
#
# For syntax checking, see:
# http://www.sxw.org.uk/computing/robots/check.html

User-agent: *
Crawl-delay: 10
# Directories
Disallow: /includes/
Disallow: /misc/
Disallow: /modules/
Disallow: /profiles/
Disallow: /scripts/
Disallow: /themes/
# Files
Disallow: /CHANGELOG.txt
Disallow: /cron.php
Disallow: /INSTALL.mysql.txt
Disallow: /INSTALL.pgsql.txt
Disallow: /INSTALL.sqlite.txt
Disallow: /install.php
Disallow: /INSTALL.txt
Disallow: /LICENSE.txt
Disallow: /MAINTAINERS.txt
Disallow: /update.php
Disallow: /UPGRADE.txt
Disallow: /xmlrpc.php
# Paths (clean URLs)
Disallow: /admin/
Disallow: /comment/reply/
Disallow: /filter/tips/
Disallow: /node/add/
Disallow: /search/
Disallow: /user/register/
Disallow: /user/password/
Disallow: /user/login/
Disallow: /user/logout/
# Paths (no clean URLs)
Disallow: /?q=admin/
Disallow: /?q=comment/reply/
Disallow: /?q=filter/tips/
Disallow: /?q=node/add/
Disallow: /?q=search/
Disallow: /?q=user/password/
Disallow: /?q=user/register/
Disallow: /?q=user/login/
Disallow: /?q=user/logout/

I believe Drupal generated this itself.

I have another Drupal site live, but it does not have a robot.txt file, and Google WebMaster Tool does not throw me such messages.

How to solve this issue? Is it safe to remove my robot.txt file?

UPDATE

I am still struggling with these strange warning messages from Google Webmaster Tools on my Drupal website:

Is robots.txt blocking important pages? 
Some important page is blocked by robots.txt.

When I click on 'important page', it points to:

http://www.mydrupalsite.com/misc/jquery.js?v=1.4.4

Why would Google want to access /misc?

4

There's nothing in your robots.txt file that is out of the ordinary and for the most part you probably want to keep all of those exclusions, checks, etc. to save on bandwidth and for security. The health message you're seeing is likely due to another issue within the site or maybe just a temporary Google hiccup that will resolve itself shortly. Are there any other messages within the Google Webmasters page that indicate other issues, ex. permission problems?

Another possibility is to submit an XML Sitemap to Google directly to tell it exactly what you are expecting it to index:

http://drupal.org/project/xmlsitemap

I've found that this resolves some Google crawl issues in the past.

3
  • No other error messages from Google WebMaster. I have already submitted a sitemap.xml generated from Drupal (no errors found by Google WebMaster too). I had indeed a crappy beta version online before I released this clean production release. Probably a Google hiccup. Apr 9 '12 at 15:26
  • Turns out it is not a hiccup, issue remains. May 2 '12 at 17:48
  • After posting something on a Google forum, the issue got solved. There was not reason Google would need to access a jQuery file in a disallowed directory. I think they edited something in their system. May 16 '12 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.