0

I have a load of content that is gone (don't ask). I would like to tell Google and other search engines that this resource is gone, please don't try to index it. This is what HTTP status code 410 looks good for.

This is great for search robots, but if it's a real person accessing the page, I'd rather they were redirected to the new content.

Luckily, all the gone URLs match a pattern.

Is it possible to have all those handled by issueing a 410, and a redirect of some sort?

(I worry that permanent redirects might affect SEO too?)

Put another way: what I want to say is "That content is gone, you should not link to it. There's new content here:..."

[EDIT: to express why this is Drupal related]

Drupal is my CMS. With any generic website I would find my own solution to present a page and the correct header. With Drupal there's usually "a module for that", sometimes an elegant way to solve a common problem, and sometimes a hideously inefficient way to square a circle known as "the Drupal Way" [joke].

Drupal has an in-built system for 404s that works great. You can set a page, and because it's got a url you can use the block module to put other useful stuff on it, e.g. a View.

So there's 2 questions:

  1. Does Drupal (or a reliably contrib'ed module) have a way to issue 410s akin to the 404 system?

  2. Does Drupal have a 302/301 redirect system that can work with path patterns (i.e. separate from node as these dead pages were from another module that has been removed)

  • This is really close to being off-topic, unless it is asking about a Drupal solution for generating 410's and/or a proper way to implement a 410 w/ redirect solution in Drupal. – mpdonadio Sep 13 '13 at 14:49
  • @MPD if it's off-topic here, can't it be migrated? Might be OK on Stack Overflow. – Mołot Sep 13 '13 at 20:15
  • @Mołot This question can be easily salvaged with some editing. – mpdonadio Sep 13 '13 at 23:55
  • @mpd I've edited and requested re-open. Can you click reopen, too? (not sure how that process works) – artfulrobot Sep 16 '13 at 9:12
  • 1
0

We just did a major redesign that include a new URL structure. We decided to to use the Redirect module as part of the plan.

One of its features is to go through the {watchdog} table, and look for Page Not Found errors (ie, 404s). It then provides a UI for adding a redirect to another URL. You also have the option for setting the status code. It will default to using a 301. From the definition:

10.3.2 301 Moved Permanently

The requested resource has been assigned a new permanent URI and any future references to this resource SHOULD use one of the returned URIs. Clients with link editing capabilities ought to automatically re-link references to the Request-URI to one or more of the new references returned by the server, where possible. This response is cacheable unless indicated otherwise.

According to Google, a 301 is the proper code to use when you want the user to goto a new page:

If you're moving that content to a new URL, you should 301 redirect the old URL to the new URL-that way when users come to the old URL looking for that content, they'll be automatically redirected to something relevant to what they were looking for.

and a 410 should be used for content that is gone forever:

If you're getting rid of that content entirely and don't have anything on your site that would fill the same user need, then the old URL should return a 404 or 410. Currently Google treats 410s (Gone) the same as 404s (Not found).

The Redirect module does not handle 410 generation, though. Otherwise, it has been a good way to handle the new scheme w/o needing to manually generate Apache rules.

  • I had this written, but was unable to post it before it got five close votes. – mpdonadio Sep 17 '13 at 0:16
  • Thanks, this looks like the right way to go. I'll look into that module. – artfulrobot Sep 17 '13 at 16:15
2

You can always output HTTP 410 header and redirect human users using JavaScript in a body:

var delay = 10000; //Your delay in milliseconds
setTimeout(function(){ window.location = URL; }, delay);

JavaScript from this answer.

Note that by rfc2616:

The 4xx class of status code is intended for cases in which the client seems to have erred. Except when responding to a HEAD request, the server SHOULD include an entity containing an explanation of the error situation, and whether it is a temporary or permanent condition. These status codes are applicable to any request method. User agents SHOULD display any included entity to the user.

Emphasis mine. So, JavaScript outputted inside HTML body should work. But of course it is not guaranteed it will.

Your content is not nodes, so you probably have implemented hook_menu. If not, you can alter existing implementation with hook_menu_alter. What you need is to:

  • Using access callback and/or access arguments, ensure user gets the access permission to deleted content. If not, he will see 403 or 404, as this is Drupal's default behaviour for unpublished / deleted data.
  • In page callback test if content is valid, deleted or never existed.
    • If valid, default to the callback you are using now.
    • If deleted, use drupal_add_http_header to add "410 gone" header as appropriate. Output nice human readable page (optionally with JavaScript redirect, search box or anything appropriate).
    • If never existed, either default to what you were using already, or implement 404 the way you implemented 410 above.

Google says:

We recommend that you always return a 404 (Not found) or a 410 (Gone) response code in response to a request for a non-existing page.

  • Thanks. So I'd set up a custom page in a module with a menu that matches this pattern, use header 410, and output text for the user. I don't want to us JS for the redirect because I'm not sure how that affects Google (and other SEO). There will be 10s or possibly 100 URLs that end up here... – artfulrobot Sep 13 '13 at 14:25
  • 1
    @artfulrobot Well, you can just output page with message, but it will be displayed under "gone" address. As far as I know (it was never officially verified) redirects with delay does not affect ranking in any way, works just like simply outputting page. On the other hand, nice message and "see this instead" link can be even more user friendly. – Mołot Sep 13 '13 at 14:34
  • 410 and 404 errors can potentially hurt a site. Google Webmaster tools will warn you about this. Assuming you have a replacement URL, a 301 is a better option in the long run, and there are Drupal ways to handle them. – mpdonadio Sep 14 '13 at 0:00
  • @MPD On the other hand, sudden redirection from many different content to one place can result in a ban, too. That's what black SEO guys was doing to suddenly boost some pages. Only safe thing not to loose content, but that's out of the question now, right? At the very least 410 is honest. Damage is done already and the only choice is how will it hurt, not if. Seems OP have decided, so I'm doing my best to help him with his decision. – Mołot Sep 14 '13 at 10:49
1

Here's a more Drupal-specific answer.

410 doesn't seem appropriate to me.

I would simply have the old URLs issue HTTP 301 redirects, which you could do with the Redirect module.

Google and other search engines would then remove the old addresses and pick up the new ones (eventually, depending on how often your site is indexed). You could even submit URL removal requests to Google for each old address if you want to try expediting the process.

If you don't want the new addresses to be indexed at all, then the new pages should use a robots 'noindex' <meta> tag, which you could do with the Metatag module.

  • 301 means "moved permanently". It's inappropriate if new content is not the same (logically) as the old, now gone one. So it's not OK in this case. – Mołot Sep 13 '13 at 20:17
  • @Mołot If the OP thinks a redirect is appropriate, then a 301 (or even a 302) is the proper solution. It also aligns closer with Google's recommendations. I had a fairly answer written about this, but couldn't post it before it got put on hold. – mpdonadio Sep 13 '13 at 23:57
  • Well, Google recommendation I linked under my answer says exactly opposite than MPD. And decision if is SEO thing. Drupal thing is how to do what OP asks to. – Mołot Sep 14 '13 at 11:05
  • For time being I've gone with 301, I'll keep eye on Google WMT to see what it complains about (it seemed v unhappy about all the 404s). The Redirect module looks really useful, thank you. – artfulrobot Sep 16 '13 at 9:11
1

I've thought a lot about using 410 (gone) for pages that we don't need. Remember backlinks are like money you never want to lose them. So only under extreme conditions should we use 410 response for gone pages.

If you want Google to not to index them and delete them all from their index exclude them in the Robots.txt( here).

So better redirect all of them using 301 to one common "Moved" page and let Google know about it( and not block using robots.txt etc). If search engines know about 301 moved then your backlink worth will be maintained.

410 response is too risky to use since once you've sent a 410 response to Google etc, if pathauto creates the same alias for another page then Google will simply ignore it. I don't think this type of error can be debugged using WMT.

I think this question is more suitable to webmasters.stackexchange.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.