0

I want to scrap location data from google. So for example; If I search for "Restaurants in Miami". I get over 14K results. I reviewed; How to import Nodes and Node Location data using Feeds module?. But that doesn't really help me with importing a query term from google.

  1. How would I be able to scrap each one of the locations that the query gives as a result and import them as a node into drupal?
  2. Are there any pre-built modules that might assist in this process?
  3. I'm I going to need to build a custom module for this use case?

UPDATE: I've found this page; http://drupal.org/project/example_web_scraper and will studying the info to see if this is what I'm looking for.

AH HA MOMENT:

  • What exactly do you want to do? – BetaRide Jul 14 '12 at 7:10
  • It's scrape, not scrap. – user7667 Jul 16 '12 at 18:14
5

Not to put a damper on things (I've written more scrapers than I care to remember and it's always fun), but such an act is specifically against Google's Terms of Service:

Do not misuse our Services, for example, do not interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.

I'm sure you'll be ok if you're doing this for fun/educational purposes (don't take my word for it though, I'm not a lawyer!), but if you make something commercial out of this you could find yourself in a tricky position.

  • I really just want to pull the location data to create nodes so I don't need to do it by hand. Doesn't Google have its MAPS API to do this very thing? – Digital fire Jul 16 '12 at 16:59
  • Also found this; code.google.com/p/gempaq/wiki/GMaps_GeoRSS_Feeds_parser. I know its for D6 but its similar to what I need to do. – Digital fire Jul 16 '12 at 17:03
  • Absolutely...but that's a stand-alone API, governed by its own set of rules (it's not related to search in their opinion, and definitely wouldn't be considered scraping, which has its own set of connotations). If you want to get location data in accordance with the terms of service then one of the APIs is the only way to go really. There's a dev version of Google Maps API v3 that might be worth a look – Clive Jul 16 '12 at 17:04
  • Again, that's fine as you're parsing data that Google have specifically shared, it's not scraping as such. Sorry if I've got hung up on that word too much :) – Clive Jul 16 '12 at 17:07
  • Interesting. Going to take a look at that later today. Thank you @Clive. – Digital fire Jul 16 '12 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.