I have a script that's taking in a ton of address data from a CSV, converting it to JSON, and along the way it hits the Google Maps API to get the latitude and longitude of each address. It then writes the whole thing to a .json file that is used by another widget. This works well for small numbers of addresses but some of the files I'm importing have hundreds of addresses. I have considered several ways around this:
- Queues: The problem with this one seems to be that I would have to save an object for each address and I don't want to completely fill up my database with address objects. I would then also have to somehow detect the completion of the queue and retrieve all those objects from the database to generate my complete json file. I worry about whether the server will deal with this well.
- Some other Google Maps API: Am I even using the correct Google Maps API for this? Is there one that can get a bunch of coordinates for a bunch of addresses in one HTTP request? I couldn't find one but that doesn't mean it's not existing.
- Feeds API (or some other core or contrib API for getting tons of data): It occurs to me that my hand-rolled solution might be circumventing some more Drupal-correct way to accomplish this task. Does Drupal offer something for making big batches of HTTP requests that I'm just missing?
So, basically my question is: what's the most Drupalish way to make and process a large number of HTTP requests from a module?