0

In my dblog occassionally I am seeing query coming probably from search engines for links as

http://mydomain.com/node/13130/delete?destination=node

I wonder how possibly search engine got idea to query that kind of links (of course those links are not exposed to anonymous users)

I have xml sitemap installed but no such links exists in the index

When I query google for that link, I see that link is present in index (of course clicking on that link leads to access denied page), and also shows

A description for this result is not available because of this site's robots.txt

Improve this question
2
  • This isn't really a Drupal question and should be posted somewhere else, by the way.
    – Alfred Armstrong
    Commented Apr 4, 2013 at 8:23
  • It thought it may be Drupal related if Drupal configuration leads to the problem, or Drupal configuration may prevent the problem.
    – NenadP
    Commented Apr 4, 2013 at 8:45

1 Answer 1

Reset to default
2

One possibility is that one or more logged-in users have Google Toolbar or something similar installed. I don't have it installed any more because I noticed when developing locally that every time I fetched a page in my browser the request was repeated by the toolbar, presumably sending it to Google.

In my experience there is no such thing as a non-public URL as far as Google is concerned. Any URL can be listed for indexing and so crawlers may attempt to fetch it from time to time. I suspect other tools may record URLs in a similar manner.

Improve this answer
1
  • Thanks Alfred for this very interesting insight! I wonder could it be prevented with more restrictive robots.txt. I will have to look into that.
    – NenadP
    Commented Apr 4, 2013 at 9:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.