Drupal, Crawlers, and Migrating from a Query string based site to a Clean URLs site

Question

I have a (potentially huge) problem that I did not anticipate. I just built a beautiful new site for the organization I work for. It is a massive improvement over the previous site. The site is also, I'd say, larger than average.

The previous site utilized query strings to navigate around the site, and in a really awful, highly customized fashion. Basically all page URLs looked something like /index.php?page=/path/to/page.php&stuff=4|5|3|1.

The new site utilizes clean URLs.

Now the problem. Crawlers from all over are rampaging through the site and causing high CPU utilization (~50% at all times). I have caching enabled as well as JavaScript and CSS aggregation.

Here's an example URI from the access log:

/?p=1%7C26%7C31%7C33%7C2%7C7&page=publications.php

In the current scheme that should just return a 404 in my opinion because that literally represents nothing. Instead it's returning 200 OK which is wrong in so many ways, but technically is right because you do in fact get the home page back. In fact, every single "page" that the crawlers look at right now 90% of the time it's getting back the homepage because it's either looking at /index.php?querystring or /?querystring where of course Drupal does nothing with the query string and just returns the homepage!

I'm not entirely sure how to communicate the correct HTTP status codes to the crawlers. Any advice is much appreciated at this time. I now understand why Google implores you to build sites with clean URL schemes...

Answer 1

You need to handle this in your httpd.conf or .htaccess before Drupal handles it.

Your best bet is to come up with some rewrite rules and 410 Gone them. Something like this

RewriteEngine On
RewriteBase /
RewriteCond %{QUERY_STRING} page=(\w+)&stuff=(\w+)  
RewriteRule ^(.*)$ - [G,L]

should be a start. You need to play with the query string to match the inbound requests.