2

I have a script that's taking in a ton of address data from a CSV, converting it to JSON, and along the way it hits the Google Maps API to get the latitude and longitude of each address. It then writes the whole thing to a .json file that is used by another widget. This works well for small numbers of addresses but some of the files I'm importing have hundreds of addresses. I have considered several ways around this:

  • Queues: The problem with this one seems to be that I would have to save an object for each address and I don't want to completely fill up my database with address objects. I would then also have to somehow detect the completion of the queue and retrieve all those objects from the database to generate my complete json file. I worry about whether the server will deal with this well.
  • Some other Google Maps API: Am I even using the correct Google Maps API for this? Is there one that can get a bunch of coordinates for a bunch of addresses in one HTTP request? I couldn't find one but that doesn't mean it's not existing.
  • Feeds API (or some other core or contrib API for getting tons of data): It occurs to me that my hand-rolled solution might be circumventing some more Drupal-correct way to accomplish this task. Does Drupal offer something for making big batches of HTTP requests that I'm just missing?

So, basically my question is: what's the most Drupalish way to make and process a large number of HTTP requests from a module?

1

It seems to me that your only option is to go with a PHP CLI script implementation. You can learn more here: http://www.php-cli.com/

In PHP in general, whenever you have a long running task, it's best to go with PHP CLI as it doesn't impose a timeout limit and have larger memory limits by default. Note here that PHP CLI and the PHP used to server web pages have different settings that you can manage separately.

Once you have a PHP CLI script in place, setup a standard Linux CRON task to hit that script say every hour or every day.

In the context of Drupal, you might want to check this post: Do I need a cron task for processing a queue? and also check the Elysia CRON module https://www.drupal.org/project/elysia_cron as they might be helpful but I think in your case you're better of with a standard PHP CLI script.

Also, keep in mind that you can call Drush commands from a PHP CLI script very easily. I have a script that runs updates and CRON automatically on all Drupal instances running on web server and it's a huge advantage. You can also look into implementing your custom Drush commands.

Good luck.

  • Yes, I know how to use Drupal's queues and have implemented them before. The part that is stumping me is that I am reluctant to do it because it requires saving every address as an object, and I don't know how/when I would write to the output file. – beth May 6 '15 at 14:11
  • I'm suggesting you use a PHP CLI script that lives outside Drupal to get the job done. You can even bootstrap Drupal from within that script if you need to save your output at the end. – Mario Awad May 7 '15 at 9:40
1

I ended up solving this by using the Geocod.io API and the PHP library built for it. It supports batch requests and allows up to 2500 free addresses per day.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.