-2

I have a node whose path alias is /example, where I test JavaScript and often share it with fellow JavaScript programmers to consult about how best to do something.

I often need to post the URL for that node in forums and Q&A sites and I fear that it would cause SEO damage to my website because this webpage looks awful, doesn't have almost any logic, and may cause search engine crawlers to falsely signal that my website is "non qualitative."

To ensure that search engines never crawl and hence never index this webpage, I have added the following line to the robots.txt file.

Disallow: /example

I am not sure if that's enough, though. Not all search engines will necessarily respect robots.txt. I could add a noindex tag to that node somehow.

How can I avoid a node is crawled and indexed by search engines in the tightest way possible?
Does Drupal have its own mechanism for that, to add a noindex tag or any other thing I might have missed?

3
  • You can add noindex headers to specific content with drupal.org/project/metatag but noindex headers are no better than robots.txt in a case like this. You would actually have to block all known crawlers with a special module to really eliminate crawlers.
    – mona lisa
    Jan 19 at 1:58
  • Or just don't do this in prod, period. Otherwise, look into content access solutions to 100% eliminate anonymous access outside of crawlers.
    – Kevin
    Jan 19 at 2:02
  • @Kevin if its not on prod, on the web, I won't be able so share the webpage for live testing with a browser console. I don't know what is the meaning of "content access solutions".
    – drupaler
    Jan 19 at 13:21

1 Answer 1

1

Yes, Drupal has a mechanism to block all crawlers in the “tightest way possible”: authentication.

2
  • Can I use Drupal core to protect a specific node with a password? I didn't find a way (or a module) to do it in Drupal 10.
    – drupaler
    Jan 19 at 14:02
  • For a specific node, drupal.org/project/protected_pages or custom code.
    – mona lisa
    Jan 19 at 14:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.