I've been requested by a client to completely block the URLs that contain a
/node/## after the hostname to prevent these locations from being indexed by search engines. My issue is that I've already defined a Page Not Found node that gets shown with error 404s.
I've already implemented a modification in the
robots.txt file to include
Disallow: /node/ Disallow: /?q=node/
to prevent the web-crawler bots. However, the URLs are still directly accessible. So in this case, the client wants two different 404 actions:
- www.clientname.com/testttt -> Drupal renders 404 page
- www.clientname.com/node/897 -> blocked completely without returning Drupal's 404 page.
Does any module allow for this kind of functionality?