How can I serve a "410 Gone" HTTP status AND redirect my removed content?

Question

I have a load of content that is gone (don't ask). I would like to tell Google and other search engines that this resource is gone, please don't try to index it. This is what HTTP status code 410 looks good for.

This is great for search robots, but if it's a real person accessing the page, I'd rather they were redirected to the new content.

Luckily, all the gone URLs match a pattern.

Is it possible to have all those handled by issueing a 410, and a redirect of some sort?

(I worry that permanent redirects might affect SEO too?)

Put another way: what I want to say is "That content is gone, you should not link to it. There's new content here:..."

[EDIT: to express why this is Drupal related]

Drupal is my CMS. With any generic website I would find my own solution to present a page and the correct header. With Drupal there's usually "a module for that", sometimes an elegant way to solve a common problem, and sometimes a hideously inefficient way to square a circle known as "the Drupal Way" [joke].

Drupal has an in-built system for 404s that works great. You can set a page, and because it's got a url you can use the block module to put other useful stuff on it, e.g. a View.

So there's 2 questions:

1. Does Drupal (or a reliably contrib'ed module) have a way to issue 410s akin to the 404 system?

2. Does Drupal have a 302/301 redirect system that can work with path patterns (i.e. separate from node as these dead pages were from another module that has been removed)

Answer 1

We just did a major redesign that include a new URL structure. We decided to to use the Redirect module as part of the plan.

One of its features is to go through the {watchdog} table, and look for Page Not Found errors (ie, 404s). It then provides a UI for adding a redirect to another URL. You also have the option for setting the status code. It will default to using a 301. From the definition:

10.3.2 301 Moved Permanently

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible. This response is cacheable unless indicated otherwise.

According to Google, a 301 is the proper code to use when you want the user to goto a new page:

If you're moving that content to a new URL, you should 301 redirect the old URL to the new URL-that way when users come to the old URL looking for that content, they'll be automatically redirected to something relevant to what they were looking for.

and a 410 should be used for content that is gone forever:

If you're getting rid of that content entirely and don't have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found).

The Redirect module does not handle 410 generation, though. Otherwise, it has been a good way to handle the new scheme w/o needing to manually generate Apache rules.

Answer 2

You can always output HTTP 410 header and redirect human users using JavaScript in a body:

var delay = 10000; //Your delay in milliseconds
setTimeout(function(){ window.location = URL; }, delay);

JavaScript from this answer.

Note that by rfc2616:

The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding to a HEAD request, the server SHOULD include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. User agents SHOULD display any included entity to the user.

Emphasis mine. So, JavaScript outputted inside HTML body should work. But of course it is not guaranteed it will.

Your content is not nodes, so you probably have implemented hook_menu. If not, you can alter existing implementation with hook_menu_alter. What you need is to:

• Using access callback and/or access arguments, ensure user gets the access permission to deleted content. If not, he will see 403 or 404, as this is Drupal's default behaviour for unpublished / deleted data.
• In page callback test if content is valid, deleted or never existed.
  • If valid, default to the callback you are using now.
  • If deleted, use drupal_add_http_header to add "410 gone" header as appropriate. Output nice human readable page (optionally with JavaScript redirect, search box or anything appropriate).
  • If never existed, either default to what you were using already, or implement 404 the way you implemented 410 above.

---

Google says:

We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page.

Answer 3

Here's a more Drupal-specific answer.

410 doesn't seem appropriate to me.

I would simply have the old URLs issue HTTP 301 redirects, which you could do with the Redirect module.

Google and other search engines would then remove the old addresses and pick up the new ones (eventually, depending on how often your site is indexed). You could even submit URL removal requests to Google for each old address if you want to try expediting the process.

If you don't want the new addresses to be indexed at all, then the new pages should use a robots 'noindex' <meta> tag, which you could do with the Metatag module.

Answer 4

I've thought a lot about using 410 (gone) for pages that we don't need. Remember backlinks are like money you never want to lose them. So only under extreme conditions should we use 410 response for gone pages.

If you want Google to not to index them and delete them all from their index exclude them in the Robots.txt( here).

So better redirect all of them using 301 to one common "Moved" page and let Google know about it( and not block using robots.txt etc). If search engines know about 301 moved then your backlink worth will be maintained.

410 response is too risky to use since once you've sent a 410 response to Google etc, if pathauto creates the same alias for another page then Google will simply ignore it. I don't think this type of error can be debugged using WMT.

I think this question is more suitable to webmasters.stackexchange.com